About the BetaSweep Scanner

BetaSweep is a desktop app that audits websites for accessibility, SEO & AI readability, performance, spelling, and HTML validation. Scans are initiated by site owners, their agencies, or authorized users.

How BetaSweep Respects Your Site

  • Follows robots.txt rules (both User-agent: * and User-agent: BetaSweep)
  • Respects Crawl-delay directives
  • Default 1-second delay between page loads
  • Does not index, cache, or redistribute content
  • Scan data is stored locally on the user's machine, not on our servers

How to Control BetaSweep via robots.txt

Add the following to your site's robots.txt file to allow BetaSweep to scan your site:

# BetaSweep - Website quality scanner for accessibility, SEO, and performance
# Learn more: https://betasweep.com/bot
User-agent: BetaSweep
Allow: /
Crawl-delay: 1

To block BetaSweep entirely:

User-agent: BetaSweep
Disallow: /

To allow specific sections only, use Allow: and Disallow: rules as needed.

How to Allow BetaSweep Through Your Firewall

If your site uses a Web Application Firewall (WAF), BetaSweep supports a custom authorization header that you can add to your allowlist:

X-BetaSweep-Token: your-token-here

Site owners set this token in BetaSweep's scan settings and add the same token to their WAF allowlist. Here's how for common providers:

Cloudflare

Create a WAF custom rule that allows requests containing the X-BetaSweep-Token header with your token value.

Sucuri

Add X-BetaSweep-Token to your trusted headers configuration.

AWS WAF

Create a rule to allow requests with the X-BetaSweep-Token header matching your token.

Other Providers

Allow requests containing the X-BetaSweep-Token HTTP header with a specific value that you control.

Contact

Questions about our scanning behavior? Contact us at support@betasweep.com.