Skip to main content

Channel99 Crawler Permissions

Mark Yatman
Updated

C99Bot is Channel99's web crawler. It fetches public HTML the same way non-rendering AI crawlers do. It does not run JavaScript.

If your site blocks unrecognized crawlers by default,  increasingly common as more sites use Cloudflare, AWS WAF, or similar tools to reduce AI crawler traffic,  you may need to explicitly allow C99Bot. This page explains how.

User-Agent

Mozilla/5.0 (compatible; C99Bot/1.0; +https://channel99.com/bot)

What it crawls

Public pages on sites Channel99 customers ask us to measure, plus URLs that AI assistants cite in those measurements. C99Bot does not crawl pages behind a login, and it does not submit forms.

Rate limits

About one request per second per host, with a short random delay between requests.

Allowing C99Bot in robots.txt

C99Bot identifies itself and respects robots.txt. To explicitly allow it:

User-agent: C99Bot
Allow: /

To block specific paths while still allowing the rest of your site:

User-agent: C99Bot
Disallow: /admin/
Allow: /

Changes to robots.txt can take some time to take effect on our end — allow up to 24 hours before assuming a change hasn't worked.

Unblocking C99Bot in your firewall or WAF

A correct robots.txt entry isn't always enough. Many sites run a Web Application Firewall (WAF) or bot-management product that blocks unrecognized crawlers at the network level, before robots.txt is ever consulted. If C99Bot is still being blocked after updating robots.txt, check your WAF configuration.

Cloudflare

  1. In your Cloudflare dashboard, go to Security → WAF.
  2. Under Custom rules, create a new rule.
  3. Set the rule condition to match on User-Agent:
    • Field: User Agent
    • Operator: Contains
    • Value: C99Bot
  4. Set the rule action to Allow, and make sure it has priority over any broader bot-blocking rules (such as Cloudflare's "Block AI Bots" managed rule).

If you use Cloudflare's Bot Fight Mode or Super Bot Fight Mode, add C99Bot to the allowlist under Security → Bots, or add an explicit WAF allow rule as above so it's evaluated before those settings.

AWS WAF

  1. In the AWS WAF console, create a string match condition for the User-Agent header, matching C99Bot.
  2. Create a rule using that condition with the action set to Allow.
  3. Associate the rule with your Web ACL, and give it higher priority than any rules that block generic or unrecognized bots.

Other firewalls or bot-management tools

Most WAF and bot-management products (including those built into hosting platforms) support allowlisting by User-Agent string. Search your provider's documentation for "allow" or "bypass" rules and add an entry matching C99Bot. If your provider also supports IP allowlisting and you need our source IP ranges, contact us using the details below.

Verifying access

You can confirm C99Bot can reach your site by checking your server or WAF logs for requests with the User-Agent string above. If you don't see any C99Bot requests within a day or two of making changes, get in touch and we'll help troubleshoot.

Identity

Requests are signed with Web Bot Auth (HTTP Message Signatures), so you can cryptographically verify that a request coming from the C99Bot user agent actually originates from Channel99. Public keys are published at:

https://www.channel99.com/.well-known/http-message-signatures-directory

Contact

Questions about C99Bot, or need help getting it unblocked? Contact us at support@channel99.com.

Comments

0 comments

Please sign in to leave a comment.