Proxies for Web Scraping: Connections, Errors and Retries

When a request goes through a proxy, the response can come from the proxy or from the website, and a status code alone does not always say which. A 407 with Proxy-Authenticate is a challenge from the proxy, and a 401 with WWW-Authenticate is a challenge from the origin server. A 429 means too many requests, and for that and for 403 you need to confirm which hop answered. This guide explains what the standards say about these signals and how to respond in a measured way, for sites you are authorized to access.

Authorization comes first

Whether you may scrape a particular site depends on its terms, any contracts you have, and applicable law. Protocol documents cannot answer that question. A proxy is a network intermediary: it carries your traffic, but it does not grant permissions on a website or platform. Technical reachability is not permission either.

Everything below assumes you already have a legitimate basis for access. Nothing here is guidance for getting around access controls.

Check robots.txt before anything else

The Robots Exclusion Protocol (RFC 9309) lets service owners state how crawlers may access their content, and crawlers are requested to honor those rules.

Two details matter for connectivity work:

• If robots.txt cannot be reached because of server or network errors, the crawler must assume complete disallow. A proxy failure while fetching robots.txt is therefore a reason to stop and fix the connection, not to continue without the rules.
• The protocol is not a security measure. RFC 9309 says access to paths should be controlled by something like HTTP Authentication. So a permissive robots.txt is not a grant of permission, and a restrictive one is not a lock.

Crawl-delay is not part of RFC 9309, so do not treat it as a standard directive.

How CONNECT tunnels work

For HTTPS targets, a client usually asks the proxy to open a tunnel with the CONNECT method, defined in RFC 9110. The details that affect debugging:

• The request target is only host:port. There is no default port, so the client must send one.
• Any 2xx response switches the connection to tunnel mode. A response that is not successful means the tunnel was not formed.
• CONNECT is meant for requests to a proxy, and most origin servers do not implement it.
• Proxies that support CONNECT should restrict it to a limited set of known ports or a configurable list of safe targets. A refusal for an unusual port can be a normal policy choice.
• Responses to CONNECT are not cacheable.

A non-2xx answer to CONNECT tells you only that no tunnel exists yet. The cause may be the proxy's authentication or policy, the proxy's own attempt to reach the destination, or another proxy further along a chain. Read the status code and any message the proxy returns before deciding which, and avoid assuming that the target site's rules are involved.

A tunnel request with placeholder values looks like this:

CONNECT TARGET_HOST:TARGET_PORT HTTP/1.1
Host: TARGET_HOST:TARGET_PORT
Proxy-Authorization: PROXY_CREDENTIALS

407, 401 and 403 are different problems

These three codes are easy to confuse, but each points to a different place.

• 407 Proxy Authentication Required is the proxy's challenge. It must carry a Proxy-Authenticate header. The client can answer by sending Proxy-Authorization, and it may repeat the request with new or replaced credentials.
• 401 Unauthorized is the origin server's challenge, sent with WWW-Authenticate. It is a separate mechanism from the proxy's.
• 403 Forbidden is what a server ought to send when it receives valid credentials that are not adequate for access. Valid credentials alone do not guarantee access to a resource.

The two challenge headers are the reliable markers of which side is asking. A bare 403 does not carry that information. Work out whether it arrived in response to a CONNECT request (the proxy answered), came back inside a forwarded request, or includes headers or a body that identify the sender.

Proxy-Authorization applies only to the next inbound proxy that demanded authentication. In a chain of proxies, the first one expecting credentials consumes them. If you route through more than one hop, check which hop is issuing the 407.

Proxy authentication can also be used to establish the authority to create a CONNECT tunnel, as in the RFC's example, where the header is sent on the CONNECT request itself.

429 and Retry-After

Status 429 comes from RFC 6585 and means the user has sent too many requests in a given amount of time. The response should include details of the condition and may include a Retry-After header saying how long to wait.

The RFC leaves several things undefined:

• How the service identifies the user and counts requests is not specified. Limits may apply per resource, server-wide, or across a set of servers, and the user may be identified by credentials or a cookie.
• Servers are not required to use 429. Under heavy load they may drop connections instead, so a reset or silent close can be a throttling signal too.
• A 429 response must not be stored by a cache.

The Retry-After value is either an HTTP-date or delay-seconds, a non-negative integer (RFC 9110). Your client should parse both forms. When a response carries the header with a 429, wait for the stated time before sending again.

A 429 is a request to slow down. Do not treat it as an obstacle to route around.

Timeouts and retries

The protocol does not set timeout values. Connection, read and total timeouts are settings of your client or library, so choose them for your own workload and test them. HTTP defines the 408 (Request Timeout) status code, but a status code does not dictate how long your client should wait.

A reasonable engineering practice is to space retries out with growing delays and some randomness (exponential backoff with jitter). That is a general practice, not a protocol requirement, and it is not a way to avoid detection or enforcement.

A few habits that fit responsible behavior:

1. Honor Retry-After whenever a response includes it.
2. Cap the number of attempts per URL and stop when the cap is reached.
3. Note whether a failure happened before a tunnel formed (no 2xx on CONNECT, 407) or in a response once requests are flowing. A 401 with WWW-Authenticate marks the origin and a 407 with Proxy-Authenticate marks the proxy. A 403 or 429 can come from either, so confirm which hop answered, since each calls for a different fix.
4. Log the status code, which hop answered, and the elapsed time for each failure.
5. Re-check your authorization if you keep receiving refusals.

SOCKS5 at a glance

SOCKS5, defined in RFC 1928, is a framework that lets TCP and UDP client-server applications use the services of a network firewall. The flow has three steps: the client negotiates an authentication method, authenticates, and then sends a relay request. The server either establishes the connection or denies it.

The RFC defines three methods: no authentication, GSSAPI, and username/password. Compliant implementations must support GSSAPI and should support username/password. The request commands are CONNECT, BIND and UDP ASSOCIATE.

Username/password authentication is specified in RFC 1929 as one of the subnegotiations in the SOCKS5 framework. HTTP status codes belong to HTTP, so the 407-style diagnosis above does not apply inside a SOCKS5 handshake. There, a failure shows up as a negotiation or authentication failure, or as a denied relay request. A client library supporting a SOCKS5 feature does not tell you what a given server permits, so test the specific method and command you need.

For related background, see the guides SOCKS4 vs SOCKS5: Authentication, DNS and UDP Compared and What Is an HTTPS Proxy and How Do SSL Proxies Work?.

Quick reference: what each signal points to

Signal	Where it may come from	First thing to check
CONNECT answered with a non-2xx status	Proxy, or its attempt to reach the destination or an upstream proxy	No tunnel was formed. Read the status and any proxy message, and check the host:port target and any port restrictions.
407 with Proxy-Authenticate	The proxy	Credentials sent in Proxy-Authorization, and which hop in a chain is challenging.
401 with WWW-Authenticate	The origin server	The site's own authentication, separate from the proxy's.
403	Origin or proxy; confirm which	Valid credentials may not be adequate for that resource, or a proxy policy may apply. Review access rights and identify the sender.
429, optionally with Retry-After	Origin or proxy; confirm which	Slow down and wait for the stated time if one is given.
Dropped or reset connection	Either side	Possible load shedding. Check logs, reduce volume and retry carefully.
robots.txt unreachable	Network or server	Assume complete disallow until it can be fetched.

For hands-on checks of connectivity, authentication and protocol, the guide How to Check a Proxy: Reproducible Connectivity, Auth, IP, DNS and Protocol Tests covers the testing side. For a broader troubleshooting overview, see Why Your Proxy Is Not Working: Errors and Fixes.

FAQ

Does a 407 mean the website rejected me?

No. A 407 is issued by a proxy and comes with Proxy-Authenticate. A 401 is the origin server's challenge and uses WWW-Authenticate. They are separate mechanisms, so fix the one that is actually answering.

Should I always retry after a 429?

Only after waiting, and only within limits you set. If the response includes Retry-After, honor it. The value is an HTTP-date or a whole number of seconds. If it is absent, slow down and cap your attempts.

Is robots.txt enough to know that scraping is allowed?

No. It tells crawlers how the owner asks them to behave, and RFC 9309 says it is not a security measure. Authorization depends on site terms, contracts and law, so check those separately.

Next step

Once you have confirmed that your use is authorized, make sure your client reads each response at the right layer: proxy, tunnel or origin. Open Proxyca in Telegram