Rate limits and crawl etiquette

How fast Wrendex crawls, how to throttle it, and the user agent it sends.

Default crawl rate

By default Wrendex makes up to five concurrent requests and no more than ten requests per second; real-browser crawls drop to three and three. The crawler honours robots.txt, including Crawl-delay directives. Most sites do not notice the audit traffic.

Throttling

From the site settings page, you can lower the number of concurrent requests per site. Sites on shared infrastructure typically run audits at the lower rate to avoid contending with production traffic during peak hours.

User-Agent

The crawler identifies as 'BigBug/1.0 (Wrendex)' on every request, and robots.txt rules for the BigBug user agent apply to it; real-browser crawls send a standard Chrome user agent. If your site gates by user agent, allow that string, or set a custom one for the site through the sites API.