About AI-bot User Agent Strings
If you want your website to be findable in AI-chat systems, you should allow these bots by user agent strings:
GPTBot/1.0— OpenAI's web crawlerChatGPT-User/1.0— ChatGPT browsingClaudeBot/1.0— Anthropic's crawleranthropic-ai/1.0— AnthropicPerplexityBot/1.0— Perplexity AIGoogle-Extended— Google's Gemini/Bard training crawlerGooglebot/2.1— General Google crawler (also feeds AI)FacebookBot/1.0— Meta's crawler (feeds AI training)meta-externalagent/1.1— Meta AI crawlerYouBot/1.0— You.com
Tips and Tricks Development Search Engine Optimization (SEO) Answer Engine Optimization (AEO) Generative Engine Optimization (GEO)
Also by me
Django Messaging
For Django-based social platforms.
Django Paddle Subscriptions
For Django-based SaaS projects.
Django GDPR Cookie Consent
For Django websites that use cookies.
