About the BetaSweep Scanner

BetaSweep is a desktop app that audits websites for accessibility, SEO & AI readability, performance, spelling, and HTML validation. Scans are initiated by site owners, their agencies, or authorized users.

What BetaSweep Scans

BetaSweep runs five scan modules, each designed to surface actionable issues that affect how a website performs for real users and search engines:

  • Accessibility—evaluates pages against WCAG 2.2 criteria (Level A, AA, or AAA) using the axe-core engine, the same engine used by browser developer tools and enterprise testing platforms.
  • SEO & AI Readiness—checks meta tags, headings, images, structured data, internal and external link health, content quality, and signals that affect how search engines and AI systems understand your site.
  • Performance—runs Lighthouse audits on each page to measure load speed, interactivity, and visual stability on both desktop and mobile device sizes.
  • Spell Check—flags potentially misspelled words across all visible page content with suggested corrections.
  • HTML Validation—identifies markup errors, deprecated elements, and structural issues that can cause rendering problems across browsers.

Scans are initiated by site owners, their agencies, or other authorized users. BetaSweep does not crawl the web on its own and only visits pages it has been directed to scan.

How BetaSweep Respects Your Site

BetaSweep is designed to be a responsible crawler. Every scan follows these principles:

  • Follows robots.txt—obeys rules for both User-agent: * and User-agent: BetaSweep. Pages disallowed in robots.txt are excluded before the scan begins.
  • Respects Crawl-delay—honors Crawl-delay directives, capped at 10 seconds. The default delay between page loads is 1 second when no directive is present.
  • No indexing or redistribution—BetaSweep does not index, cache, or redistribute any content it encounters. Scanned data is used solely to generate the scan report.
  • Local-only storage—scan data is encrypted and stored on the user's own machine. It is never uploaded to BetaSweep's servers or shared with third parties.
  • Identifies itself clearly—all requests use the user agent string BetaSweep/1.0.0 (+https://betasweep.com/bot) so you can identify the scanner in your access logs.

How to Control BetaSweep via robots.txt

You can control exactly which parts of your site BetaSweep is allowed to scan by adding rules to your robots.txt file. To allow BetaSweep full access:

# BetaSweep - Website quality scanner for accessibility, SEO, and performance
# Learn more: https://betasweep.com/bot
User-agent: BetaSweep
Allow: /
Crawl-delay: 1

To block BetaSweep from scanning your site entirely:

User-agent: BetaSweep
Disallow: /

You can also allow or block specific sections. For example, to allow scanning of your public pages but block admin or staging areas, use Allow: and Disallow: rules as needed. BetaSweep checks robots.txt before every scan and filters out any disallowed URLs automatically.

How to Allow BetaSweep Through Your Firewall

Some websites use a Web Application Firewall (WAF) or bot protection service that may block automated scanners, including BetaSweep. If your scan results are incomplete or show blocked pages, you can allowlist BetaSweep using a custom authorization header.

BetaSweep sends a configurable header with every request during a scan:

X-BetaSweep-Token: your-token-here

To set this up, choose a token value (any passphrase you control), enter it in BetaSweep's scan settings, and add a matching allowlist rule in your WAF. Here's how for common providers:

Cloudflare

Go to Security → WAF → Custom Rules. Create a rule that matches requests where the X-BetaSweep-Token header equals your token value, and set the action to Allow. This lets BetaSweep bypass Cloudflare's bot protection while keeping all other protections in place.

Sucuri

In the Sucuri dashboard, navigate to your site's firewall settings and add X-BetaSweep-Token with your token value to the trusted headers or allowlist configuration.

AWS WAF

Create a custom rule in your AWS WAF web ACL that inspects the X-BetaSweep-Token header. Set the rule to allow requests where the header value matches your token.

Other Providers

Most WAF and bot protection services support header-based allowlisting. Create a rule that allows requests containing the X-BetaSweep-Token HTTP header with the specific value you've configured. If your provider doesn't support custom header rules, you can allowlist BetaSweep's user agent string instead: BetaSweep/1.0.0.

Contact

If you have questions about BetaSweep's scanning behavior, need help configuring your WAF allowlist, or want to report an issue, contact us at support@betasweep.com.