Allowing BetaSweep to Scan Your Site

Getting Set Up

Depending on your server and/or CMS, you may need to adjust one or more of the following to allow BetaSweep to scan your site.

• Robots.txt

  You can control exactly which parts of your site BetaSweep is allowed to scan by adding rules to your robots.txt file. To allow BetaSweep full access:

  # BetaSweep - Website quality scanner for accessibility, SEO, and performance
  # Learn more: https://betasweep.com/bot
  User-agent: BetaSweep
  Allow: /
  Crawl-delay: 1

  To block BetaSweep from scanning your site entirely:

  User-agent: BetaSweep
  Disallow: /

  You can also allow or block specific sections. For example, to allow scanning of your public pages but block admin or staging areas, use Allow: and Disallow: rules as needed. BetaSweep checks robots.txt before every scan and filters out any disallowed URLs automatically.

• Firewall (WAF)

  Some websites use a Web Application Firewall (WAF) or bot protection service that may block automated scanners, including BetaSweep. If your scan results are incomplete or show blocked pages, you can allowlist BetaSweep using a custom authorization header.

  BetaSweep sends a configurable header with every request during a scan:

  X-BetaSweep-Token: your-token-here

  To set this up, choose a token value (any passphrase you control), add a matching allowlist rule in your WAF, and enter the token in BetaSweep. You can enter your token in two places: when BetaSweep detects a firewall during the pre-scan, it will pause and prompt you to enter a token right there, or you can enter it ahead of time in the scan settings under General Settings.

  Here's how to set up the allowlist rule for common providers:

  Cloudflare

  Go to Security → WAF → Custom Rules. Create a rule that matches requests where the X-BetaSweep-Token header equals your token value, and set the action to Allow. This lets BetaSweep bypass Cloudflare's bot protection while keeping all other protections in place.

  Sucuri

  In the Sucuri dashboard, navigate to your site's firewall settings and add X-BetaSweep-Token with your token value to the trusted headers or allowlist configuration.

  AWS WAF

  Create a custom rule in your AWS WAF web ACL that inspects the X-BetaSweep-Token header. Set the rule to allow requests where the header value matches your token.

  Other Providers

  Most WAF and bot protection services support header-based allowlisting. Create a rule that allows requests containing the X-BetaSweep-Token HTTP header with the specific value you've configured. If your provider doesn't support custom header rules, you can allowlist BetaSweep's user agent string instead: BetaSweep/1.0.0.

• CMS Security Plugins

  Some content management systems have their own security plugins that block automated requests independently of any external firewall. If you've configured a WAF allowlist and BetaSweep is still being blocked, the block may be happening at the CMS level.

  You can usually resolve this by allowlisting BetaSweep's user agent (BetaSweep/1.0.0) or the X-BetaSweep-Token header in your CMS security plugin's settings.

  WordPress

  Wordfence: Go to Wordfence → Firewall → Manage Firewall → Allowlisted URLs/User Agents. Add BetaSweep to the user agent allowlist.

  Sucuri Security (plugin): In the Sucuri plugin settings, navigate to the firewall section and add BetaSweep to the allowlisted user agents. This is separate from the Sucuri cloud WAF.

  iThemes Security / Solid Security: Go to the security settings and add BetaSweep to the bot allowlist or disable the bot detection temporarily during the scan.

  Craft CMS

  Security plugins like Blitz or custom middleware may block automated requests at the application level. Check your plugin settings for a user agent or IP allowlist and add BetaSweep.

  Drupal

  Modules like Security Kit or Shield may block unfamiliar user agents. Check the module's configuration for an allowlist option and add BetaSweep.

  Other CMS Platforms

  If your CMS has a security plugin or built-in bot protection, look for a user agent allowlist in its settings and add BetaSweep. If the plugin supports header-based rules, you can allowlist the X-BetaSweep-Token header instead.

• Still Blocked?

  If you've set up a WAF token and BetaSweep is still being blocked, check the following:

  Header Name Mismatch

  BetaSweep sends X-BetaSweep-Token. Make sure your WAF rule is checking for this exact header name. Some WAF dashboards normalize header names to lowercase (x-betasweep-token)—this is fine, since HTTP headers are case-insensitive.

  Token Value Mismatch

  The value in your WAF rule must exactly match what's entered in BetaSweep. Extra spaces, different casing, or quotes around the value can cause a mismatch.

  Wrong Rule Action

  The WAF rule needs to be set to Allow or Skip—not Log, Challenge, or Block. In Cloudflare specifically, the action should be "Skip" with the relevant security features selected.

  Rule Priority

  Another WAF rule with higher priority might be blocking the request before your allowlist rule is evaluated. Check whether your rule is showing any hits in your WAF dashboard—if it shows zero hits, the request is being blocked before it reaches your rule.

  Bot Fight Mode (Cloudflare)

  If you use Cloudflare and your WAF rule shows no hits, Bot Fight Mode (or Super Bot Fight Mode) may be blocking requests before custom WAF rules are evaluated. You can either temporarily disable Bot Fight Mode under Security → Bots, or create a Skip rule that bypasses Bot Fight Mode when the X-BetaSweep-Token header is present.

  Server or Hosting-Level Block

  The block may not be coming from your WAF at all. IP-based blocking, geo-restrictions, or server-level security (like a CMS security plugin) can reject requests before your WAF rules apply. See the CMS Security Plugins section above.

If you have questions about BetaSweep's scanning behavior, need help allowing BetaSweep through your firewall or CMS, or want to report an issue, contact us at support@betasweep.com.