Black Friday Sale: Save 50% on Pro Plugins & Books w/ code BF2026
Web Dev + WordPress + Security
Tag: robots
33 posts

Ultimate Block List to Stop AI Bots

More than you might think, AI (Artificial Intelligence) and ML (Machine Learning) bots are crawling your site and scraping your content. They are collecting and using your data to train software like ChatGPT, OpenAI, DeepSeek, and thousands of other AI creations. Whether you or anyone approves of all this is not my concern for this post. The focus of this post is aimed at website owners who want to stop AI bots from crawling their web pages, as much as […] Continue reading »

Redirect Stupid Bots to Existing Resources

In case you hadn’t noticed, I’m on another one of my posting sprees. Going through the past year’s worth of half-written drafts and collected code snippets, and sharing anything that might be useful or interesting. Here is a bit of .htaccess that brings together several redirection techniques into a singular plug-&-play code snippet. Continue reading »

Target User Agents and Reduce Spam via robots.txt

Your website’s robots.txt file probably contains some rules that tell compliant search engines and other bots which pages they can visit, and which are not allowed, etc. In most of the robots.txt files that I’ve looked at, all of the Allow and Disallow rules are applied to all user agents. This is done with the wildcard operator, which is written as an asterisk *, like this: User-agent: * This site’s robots.txt file provides a typical example. All of the allow/disallow […] Continue reading »

WordPress Plugin: Disable WP Robots

WordPress 5.7 features a new Robots API that provides filter-based control over the robots meta tag. So if your site is running WordPress 5.7 or better, you will notice a new <meta /> tag included in the <head></head> section of your web pages. By default, the meta tag added by WordPress has a value of max-image-preview:large, which is fine IF it is the only robots meta tag on the page. If your site already has its own meta robots tag, […] Continue reading »

Code Snippets to Customize WordPress Sitemaps (Complete Guide)

By now most have heard about the WP Sitemaps feature introduced in WordPress version 5.5. From what I’ve read most existing sites that needed a sitemap already had one via one of the many free sitemap plugins. But for new WordPress sites going forward, having all the sitemap code in the WordPress core now means that new sites have the option of rolling with the default WordPress sitemaps, or use a dedicated plugin to do the job. This post is […] Continue reading »

About the Auto-Generated WP Sitemaps and How to Disable Using Simple Code or a Free Plugin

WordPress 5.5 and beyond features built-in sitemaps that are enabled by default. For new users and sites this may be a good thing. Now users don’t have to bother with thinking about how to implement a sitemap. Like with Privacy control, WordPress just does it for you automagically. BUT for the millions of sites that already have a sitemap thanks to any of the excellent and free sitemap plugins — that’s like maybe 5–10 million websites — well congratulations you […] Continue reading »

All the little .txt files you can put in the root directory of your website

The ones I know of: ads.txt humans.txt robots.txt security.txt This site makes use of robots.txt and humans.txt. I don’t need ads.txt because 3rd-party ads aren’t currently running on the site, and security.txt seems not necessary as the site’s contact form is easy enough for anyone to find. Continue reading »

Blackhole for Bad Bots – Quick Start

Welcome to the Quick Start Guide for the standalone PHP version of Blackhole for Bad Bots. This post basically is a condensed summary of the original Blackhole tutorial. So if you are new to the concept of blocking bad bots, check out the original tutorial. Otherwise, for those that are familiar, the following guide should simplify things and help you get started with Blackhole as quickly as possible. Continue reading »

Worst IPs: 2016 Edition

A little late this year, but following tradition here is my list of the absolute worst IP addresses from 2016. All in nice numerical order for easy crunching. These IPs are associated with all sorts of malicious activity, including exploit scanning, email harvesting, brute-force login attacks, referrer spam, and everything in between. Really obnoxious stuff that degrades your site’s performance and potentially threatens security. Continue reading »

How to Block Baidu Bot

A user of my 6G Firewall recently asked how to block the “baidu” bot from accessing their site. This post explains why Baidu is not blocked in 6G and provides a quick .htaccess technique to deny it (or anything claiming to be it) access to your site. Continue reading »

Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
.htaccess made easy: Improve site performance and security.
Thoughts
Fun tool for visualizing how different internet protocols work.
Working on a complete redesign of Perishable Press :)
The Office is Cheers with more irony.
More fine-grained control of macOS screen brightness: hold down Option + Shift before pressing either of the brightness keys. You get 64 increments instead of the usual 16.
REST Pro Tools featured in David McCan’s new video on locking down WordPress.
Gonna try exercise.
Launching my new plugin, REST Pro Tools 🛠️ Granular control of the WP REST API.
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.