Default crawl rate
Wrendex's default crawl rate is two concurrent requests with a one-second delay between bursts. The crawler honours robots.txt and respects Crawl-delay directives. Most sites do not notice the audit traffic.
Throttling
From the site settings page, you can lower the crawl rate per site. Sites on shared infrastructure typically run audits at the lower rate to avoid contending with production traffic during peak hours.
User-Agent
The crawler identifies as 'Wrendex/1.x (+https://wrendex.com/docs)' on every request. Sites that gate by user agent should allow this string. The IP range used for crawls is published on request to security@wrendex.com.
Allow-listing
Some staging environments use IP allow-listing or basic auth. Wrendex supports basic auth headers per site; enterprise plans can configure a static IP allow-list for our crawler.