How to Protect Your Websites from Malicious Bots with Imunify360 Anti-Bot Protection

This document provides a comprehensive guide to managing bot protections and custom rules with Imunify360. If you are a website owner without root access, you can ask your server administrator to implement this security feature. If you are one of our private VM or bare-metal server owners, let our technical support team implement this for your server by visiting our support desk..

Table of Contents

• Enabling and Disabling Anti-Bot Protection
• Imunify360, Fake Crawlers, and WAF Rules
• Custom Rule Solutions
• Logs and Debugging

Bots are software programs that can automate tasks on the internet. They are often used for legitimate purposes, such as crawling websites to index them for search engines. However, bots can also be used for malicious purposes, such as attacking websites or stealing data.

Imunify360 Anti-Bot Protection is a feature of the Imunify360 security suite that helps to protect websites from malicious bots. It uses a combination of methods to identify and block bots, including:

• A JavaScript challenge that most bots cannot pass.
• A blacklist of known malicious bots.
• A whitelist of allowed bots, such as search engine crawlers.

The JavaScript challenge is a simple puzzle that is displayed to visitors to the website. If the visitor is a bot, they will not be able to solve the puzzle and will be blocked from accessing the website. Legitimate visitors, who have JavaScript enabled in their browsers, will be able to solve the puzzle and continue browsing the website.

The blacklist of known malicious bots is a list of IP addresses or hostnames that are known to be used by bots to attack websites. If a bot attempts to access the website from an IP address or hostname that is on the blacklist, it will be blocked.

The whitelist of allowed bots is a list of IP addresses or hostnames that are allowed to access the website, even if they are not able to solve the JavaScript challenge. This list can be used to allow search engine crawlers and other legitimate bots to access the website.

Imunify360 Anti-Bot Protection is a powerful tool that can help protect websites from a variety of malicious bots. It is easy to configure and use, and it can be a valuable addition to any website security stack.

Here are some of the benefits of using Imunify360 Anti-Bot Protection:

• It can help to protect your website from a variety of malicious bots, including password-guessing bots, vulnerability scanners, and malware uploaders.
• It is easy to configure and use.
• It is a cost-effective way to protect your website.
• It is compatible with a wide range of web servers and applications.

To get started with Imunify360 Anti-Bot Protection, you will need to sign up for an Imunify360 account. Once you have an account, you can enable Anti-Bot Protection in the Imunify360 dashboard.

You can also configure Anti-Bot Protection to meet your specific needs. For example, you can choose to whitelist certain IP addresses or hostnames, or you can adjust the settings for the JavaScript challenge.

Enabling and Disabling Anti-Bot Protection

Enable Anti-Bot Protection (Splash Screen)

imunify360-agent config update '{"WEBSHIELD": {"splash_screen": true}}'

Disable Anti-Bot Protection (Splash Screen)

imunify360-agent config update '{"WEBSHIELD": {"splash_screen": false}}'

Imunify360, Fake Crawlers, and WAF Rules

Imunify360 has built-in security mechanisms to filter out illegitimate crawler bots. The Web Application Firewall (WAF) protections against bots are advanced features that we do not recommend disabling.

For a bot to be allowed, it must pass multiple checks:

1. IP must be in the ipset whitelist.
2. User-Agent must match a list of good bots (e.g., Googlebot).
3. Additional checks against good-bots.v2.rbl.imunify.com.

Custom Rule Solutions

General Steps for Custom Rules

1. Check Current Configuration: Use the following command to check the current modsec configuration files.

   apachectl -t -D DUMP_INCLUDES | grep modsec

2. Choose a File: Pick a file from the Imunify360 rules directory to avoid conflicts during auto-updates.

Solution 1: Custom Rule for Googlebot

1. Edit Configuration File

   sudo vi /etc/apache2/conf.d/modsec/modsec2.user.conf

2. Add Custom Rule: Add the following rule to allow specific Googlebot IPs.

   SecRule REMOTE_ADDR "@ipMatchFromFile crawlers-iplist-custom.data" "id:88999901,phase:2,t:none,log,pass,ctl:ruleRemoveById=33311,severity:5,msg:'Custom WAF: Allow Googlebot crawler from custom-crawlers-iplist||T:APACHE||User-Agent:%{REQUEST_HEADERS.User-Agent}||MV:%{MATCHED_VAR}',chain, tag:'service_i360'" SecRule REQUEST_HEADERS:User-Agent "@contains Googlebot/2.1" "t:none"

3. Create IP List File: Create a file named crawlers-iplist-custom.data and add the IPs you want to allow.

   sudo vi crawlers-iplist-custom.data

Solution 2: Custom Rule for Bingbot

1. Edit Configuration File

   sudo vi /etc/apache2/conf.d/modsec/modsec2.user.conf

2. Add Custom Rule: Add the following rule to allow Bingbot based on certain conditions.

   SecRule REQUEST_URI "@contains sitemap" "chain,id:88722311,phase:1,t:none,t:lowercase,pass,severity:2,msg:'IM360 WAF: Found bing crawler ||T:{}||',tag:'service_i360_custom'" SecRule REQUEST_HEADERS:User-Agent "@contains Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "t:none"

3. (Optional) Update Configuration: Another option is switching the rule to block mode which will instantly block the request and return a 403 HTTP denied status code.

   imunify360-agent config update '{"MOD_SEC_BLOCK_BY_CUSTOM_RULE": {"88722311": {"check_period": 60,"max_incidents": 15}}}'

Logs and Debugging

1. Restart Web Server

   sudo /usr/sbin/apachectl configtest

   sudo /scripts/restartsrv_httpd

2. Check Logs: You can find logs at /var/logs/imunify360/console.log to confirm the rules are working as expected.

Imunify360 Anti-Bot Protection is a powerful tool that can help protect your servers and website from malicious bots and is a great addition to any website security stack. If you are looking for a way to protect your web servers/sites from malicious bots, then Imunify360 Anti-Bot Protection is a good option to consider.

Want to use Imunify360? Ask us about it by either chatting with us live (if we are available for live chat) or sending an email to team+security@webhostingmagic.com