Skip to main content
GlossaryAI Crawlers

ClaudeBot

Anthropic's crawler that fetches pages for Claude's retrieval and training.

Also known asClaudeBotanthropic-aiClaude-Web

ClaudeBot (and its sibling identifiers `anthropic-ai` and `Claude-Web`) fetches pages on behalf of Claude. Like GPTBot, it respects robots.txt and honours Crawl-Delay headers. Anthropic publishes the IP ranges and user-agent strings in its trust center.

Claude is notable for weighting structured data heavily — pages with clean JSON-LD and an explicit llms.txt presence are disproportionately cited. Allow-listing ClaudeBot and shipping Organization + Product schema moves the needle fast.

Related terms

All terms