{"id":3624,"date":"2025-05-04T13:30:15","date_gmt":"2025-05-04T11:30:15","guid":{"rendered":"https:\/\/iconsult.agency\/blog\/?p=3624"},"modified":"2025-05-04T13:30:15","modified_gmt":"2025-05-04T11:30:15","slug":"how-to-block-unwanted-robots-in-apache-and-nginx-and-robots-txt","status":"publish","type":"post","link":"https:\/\/iconsult.agency\/blog\/en\/how-to-block-unwanted-robots-in-apache-and-nginx-and-robots-txt\/","title":{"rendered":"How to block unwanted robots in Apache and Nginx and Robots.txt"},"content":{"rendered":"

This is a translation of an article from Builder Society. Link to the source at the end.<\/p>\n\n\n\n

<\/p>\n\n\n

To avoid sending people to my website moneyoverethics.com for the latest robots, I decided to just create a working\/updated version here. I recommend using the following browser add-ons to check if you are actually blocking the bots you want to block. \ud83d\udc47
<\/p>\n\n\n\n\n\n\n

You can do this with the User Agent Extension:
for firefox:<\/strong> https:\/\/addons.mozilla.org\/en-US\/firefox\/addon\/user-agent-switcher\/<\/a>
for chrome:<\/strong> https:\/\/chrome.google.com\/webstore\/detail\/user-agent-switcher-for-c\/djflhoibgkdhkhhcedjiklpkjnoahfmg<\/a><\/p>\n\n\n

If you don't want to mess around with .htaccess and codes, you can use a plugin for this task.<\/p>\n\n\n

There are free plugins and there are paid plugins, such as Spyder Spanker.
You can use Link Privacy from the free ones.
<\/p>\n\n\n

Note.<\/strong><\/span> Don't just blindly copy these files without reviewing them and double checking to make sure you're not blocking the bots that should be climbing your site. For example, these files block 'curl', 'python', 'perl' and even 'SEMRush' - if you use bots or services blocked in these files, you are not only blocking your competitors from using these services to gather information about your domain, but you are also blocking your own attempts. I block everything!<\/p>\n\n\n\n

Table of Contents<\/div>
show<\/div><\/div>