About Rejecting Particular Bots

The robots.txt file relies on the goodwill of the bots. If you have some undesirable bots still accessing your website, you can block their requests based on the User Agent. This is how to do that in nginx configuration:

map $http_user_agent $is_undesirable_bot {
    default 0;
    ~evil-spider 1;
    ~evil-crawler 1;
    ~evil-bot 1;
}

server {
    # ...
    if ($is_undesirable_bot) {
        return 403;
    }
}