Scrapy’s HTTP proxy middleware can apply a proxy to an individual request through request.meta['proxy']. Start with one explicit request, then propagate the setting to subsequent requests deliberately. A proxy configured for the first request should not be assumed to cover every request your spider later creates.
This guide uses HTTP proxy authentication and a single diagnostic URL. It is a configuration example for a permitted workflow, not a recipe for bypassing access restrictions. For general networking context, read the proxy checking guide.
Starting point: one spider, one endpoint and one request. Keep credentials out of the spider file. Disable retries for the first diagnostic so that one failure is easy to interpret. The illustrative timeout and concurrency settings are not a universal production configuration.
Create a minimal proxy request
The Scrapy HttpProxyMiddleware documentation describes request-level proxy configuration. The example below reads an authenticated proxy URL from an environment variable and asks ipify to report the observed address.
import os
import scrapy
class ProxyCheckSpider(scrapy.Spider):
name = "proxy_check"
custom_settings = {
"CONCURRENT_REQUESTS": 1,
"RETRY_ENABLED": False,
"DOWNLOAD_TIMEOUT": 30,
}
async def start(self):
yield scrapy.Request(
"https://api.ipify.org?format=json",
meta={"proxy": os.environ["PROXY_URL"]},
callback=self.parse,
errback=self.failed,
)
def parse(self, response):
yield {"observed_ip": response.json()["ip"]}
def failed(self, failure):
self.logger.error(
"Proxy diagnostic failed: %s",
failure.type.__name__,
)Save this as proxy_check.py and run scrapy runspider proxy_check.py. This example uses the asynchronous start() entry point documented for current Scrapy releases. If your project pins an older version, use its matching spider documentation rather than combining incompatible lifecycle examples.
The request goes to an external IP lookup service. Use an endpoint you control if you need to keep diagnostic traffic within your own infrastructure. Do not put a real password into a shared terminal transcript.
Format the endpoint correctly
For authenticated HTTP proxy access, a URL commonly takes the form http://username:password@host:port. Obtain the actual scheme and endpoint from your order. Encode special characters in the username and password as URL components before building the URL; do not encode the whole URL as one string.
Keep destination credentials separate. The spider’s access to a website, including any account login, is not supplied by proxy authentication. Never add the proxy password as a query parameter on the target URL.
Apply the proxy to later requests
Once the diagnostic succeeds, add one permitted destination request. When constructing that request, give it the same explicit meta['proxy'] value if that is the intended route. For a larger project, centralizing this decision in your own downloader middleware can be easier to maintain than repeating it in callbacks.
Avoid copying an entire response’s meta dictionary into every new request without understanding its contents. Treat proxy selection as a named part of your application’s configuration. Record the destination hostname and a non-secret endpoint label so that logs remain useful without exposing credentials.
Understand configuration precedence
Scrapy settings can come from the command line, spider configuration and project settings. The settings reference documents that precedence. If the running behaviour differs from your file, look for a higher-priority setting before editing several unrelated components.
For a diagnostic, reduce the number of moving parts. Keep retries off until the first result is understood. Later, choose download timeout, concurrency and delay values for the permitted workload. These settings control your crawler; they do not establish the target’s permission or published rate limits.
Classify the result before changing the proxy
A returned IP means the diagnostic endpoint observed a request. A callback failure might instead involve response parsing or spider code. A downloader error concerns a different stage. Keep these stages distinct so that a JSON parsing error is not reported as an unavailable proxy.
For a target response such as 403 or 429, identify the responding service and follow its access rules. Our scraping connectivity guide explains the difference between authentication, destination responses and responsible retry handling.
Frequently asked questions
Does this configure SOCKS proxies?
No. This walkthrough uses Scrapy’s HTTP proxy middleware. A SOCKS endpoint requires a separately supported transport or integration; changing the displayed protocol name is not enough.
Why did the first request work but later ones go directly?
Inspect how those later requests are created. Make the route explicit for each relevant request or implement a central policy in middleware.
Where do I obtain the proxy details?
Open Proxyca, choose a plan and an available country, and use the issued endpoint and credentials. Open Proxyca in Telegram.