About Rejecting Particular Bots
The robots.txt file relies on the goodwill of the bots. If you have some undesirable bots still accessing your website, you can block their requests based on the User Agent. This is how to do that in nginx configuration:
map $http_user_agent $is_undesirable_bot {
default 0;
~evil-spider 1;
~evil-crawler 1;
~evil-bot 1;
}
server {
# ...
if ($is_undesirable_bot) {
return 403;
}
}
And here's how you can block requests from undesirable bots in .htaccess file with Apache:
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} (evil-spider|evil-crawler|evil-bot) [NC]
RewriteRule (.*) - [F,L]
# ...
Django/Python Consulting
If you have a specific Django challenge or integration you'd like to solve, I'd be happy to help. Book a free 30-minute call to discuss your project, see if we're a good fit, and explore the best approach for your needs. After the call, you'll receive a tailored cost estimate based on what we discuss.
Also by me
Django Messaging
For Django-based social platforms.
Django Paddle Subscriptions
For Django-based SaaS projects.
Django GDPR Cookie Consent
For Django websites that use cookies.
