Bot-trap
| tbd @ |
|---|
Bot-trap - A Bad Web-Robot Blocker
This package is meant to force web crawlers to follow the rules in the robots.txt file by denying access to any that ignore them. Googlebot and other well-behaved robots are not affected.
The software requirements are Apache and PHP, but the principles would work with any web server setup.
It is advisable to also use a web application firewall and an intrusion prevention software framework like fail2ban to protect the web server - bot-trap is not a substitute for either of those!
Possible Usage Scenarios
If you are required by law to provide a contact page with a valid email in it, this package can protect against email-harvesting robots - whether they follow the rules in robots.txt or not:
- Exclude your contact page in robots.txt.
- Email harvesting bots that follow robots.txt won't get your email.
- Email harvesting bots that don't follow robots.txt will probably find the bot-trap first, get banned and thus probably won't get your email.
Similarly, you could exclude any content you want in robots.txt, but keep in mind that this content will not be read by well-behaved bots like search engine crawlers either.
Bot-trap will also partially protect against web scrapers who do not follow the rules in robots.txt, because sooner or later they will find the bot-trap and get banned.
But if a large scraper botnet targets your website, some of the IP-addresses may get banned, but others will keep requesting pages. In this case, you could use .htaccess rewrite or if rules to redirect requests typical for the botnet to the bot-trap to increase ban-rates (see section "#Redirecting traffic to bot-trap"). This could fill up your htaccess rather quickly though, so it would be advisable to ban the IP-ranges where most requests are coming from in the web application firewall.
Demo
If you want to try it out, go to links.blauparadiesvogel.de/bot-trap/index.php. You will receive a 403 page telling you that you have been blocked, and you will be able to unblock yourself following the link provided by the 403 page.
How It Works
- Place a transparent tiny image in your web pages, which links to the bot-trap. Regular visitors don't even see this link, but bots do, since they process the code of the page.
- The robots.txt file that tells bots not access the directory of the bot-trap, thus well-behaved bots do not follow the tiny image link.
- Bots that ignore the robots.txt rules access the bot-trap and trigger the php-Script which adds their IP-address to the deny list in .htaccess, so every future request from the address will be denied.
- If a regular user is denied, the 403 page will give them the opportunity to unblock themselves by providing the correct answer to the question on the 403 page.
Installation
- Download the tarball from my nextcloud
- Unpack the tarball in your web page root directory:
# tar -xzf bot-trap.tar.gz - The bot-trap directory should be owned by the same user or group as the web server (www-data on Debian GNU/Linux); the web server user needs read access to the bot-trap directory, but it doesn't have to have write access to it.
- If you already have a robots.txt file, add the lines found in "bot-trap/docs/robots.txt.example". If you do not have such a file, bot-trap will create it when it runs.
- If you already have a .htaccess file in your root directory, add the lines found in "bot-trap/docs/htaccess-root-example". Make sure "</RequireAll>" is the last line. If you do not have such a file, bot-trap will create it when it runs.
- bot-trap will also create a deny file and a queue file in the web page root directory when it runs. You can configure the names of the files in "includes/settings.php". They must be writable by the web server user.
- Make sure .htaccess controls are allowed in your Apache configuration (especially the "AllowOverride" directive). This allows bot-trap to ban IP-addresses using the htaccess mechanism.
- To make it a bit harder for bots to specifically avoid the bot-trap while ignoring the rest of robots.txt, you should change the name of the folder. Simply rename the folder and follow the instructions in "includes/settings.php".
- You might want to rewrite the messages users receive in forbid.php, index.php and unban.php. A german language version can be found in "docs/de".
- By default, bot-trap writes every IP directly into .htaccess. On servers which are hit by large botnets this could impact performance. You could change $number_hits in "includes/settings.php" to mitigate this problem.
- Add the tiny image link to the bot-trap to your web pages:
<a href="/bot-trap/index.php <img src="bot-trap/pixel.gif" border="0" alt=" " width="1" height="1"/></a> - Make sure .htaccess and all other files have the correct permissions and ownership for your site.
- Test the trap for yourself:
- Visit the URL of the trap
- make sure you are denied access on your next request to your website
- visit the 403 page and provide the correct word
- make sure you are no longer denied access on your next request to your website
- If everything checks out, bot-trap is successfully installed!
Warning
bot-trap writes to .htaccess. If there is an error, your whole website will be unaccessible. Always keep a working copy of your .htaccess at hand.
If you want to prevent specific IPs from getting banned if they visit the bot-trap, you will have to use rewrite rules in htaccess to prevent those requests from reaching the bot-trap, or use "if"-statements to grant access (see section "#Redirecting traffic to bot-trap").
Redirecting traffic to bot-trap
There may be scenarios in which you want to redirect specific requests made to your website to the bot-trap, or prevent specific requests made to the bot-trap from reaching the bot-trap. In these cases you can use mod_rewrite rules in your .htaccess.
A few examples:
If you want to redirect all requests without user agent to the bot-trap:
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule ^.*$ https://your.website.example/bot-trap/index.php [R=301,QSD,END]
Alternatively, you could use an "if"-statement to simply deny access:
<If "%{HTTP_USER_AGENT} == ''
Require all denied
</If>
If you want to prevent a specific IP or IP-range from getting blocked by the bot-trap, you could rewrite their access to the bot-trap to the standard 403 error page on your system:
RewriteCond %{REMOTE_ADDR} ^192\.168\.1\.0$
RewriteCond %{REQUEST_URI} /bot-trap/index\.php [NC]
RewriteRule ^.*$ https://your.website.example/errors/403.php [R=301,QSD,END]
RewriteCond %{REMOTE_ADDR} ^192\.168\.1\.
RewriteCond %{REQUEST_URI} /bot-trap/index\.php [NC]
RewriteRule ^.*$ https://your.website.example/errors/403.php [R=301,QSD,END]
Alternatively, you could use an "if"-statement to grant access:
<If "%{REMOTE_ADDR} == '192.168.1.0'
Require all granted
</If>
<If "-R '192.168.1.0/24'
Require all granted
</If>
Bot-trap and MediaWiki
In MediaWiki, there is an easy way to deploy the link to the bot-trap in every page's footer; just add the following code to the "localsettings.php" (replacing your URL) and create the page MediaWiki:bot-trap-desc with empty content:
use MediaWiki\Html\Html;
$wgHooks['SkinAddFooterLinks'][] = function ( Skin $skin, string $key, array &$footerlinks ) {
if ( $key === 'places' ) {
$footerlinks['nogo'] = Html::rawElement( 'a',
[
'href' => 'https://yourwebsite.example/bot-trap/index.php',
'rel' => 'noreferrer noopener' // not required, but recommended for security reasons
],
$skin->msg( 'bot-trap-desc' )->escaped() // test-desc is an i18n message of the text
);
};
};
License
bot-trap is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
bot-trap is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
Contact and Notes
Please feel free to contact me at Wiki Aventurica if you have any suggestions or if you have found a bug. The most recent version of this bot-trap is found at my nextcloud, this description page at my linkwiki.
This rewritten version of bot-trap uses the new .htaccess syntax of Apache 2.4.
The original bot-trap was written by Daniel Webb, who was inspired by the bottrap at www.kloth.net
