Book Sale! Code WP2025 takes 20% OFF our Pro Plugins & Books »
Web Dev + WordPress + Security
Tag: robots
33 posts

About the Robots.txt File

About the Robots Exclusion Standard: The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. The information specifying the parts that should not be accessed is specified in a file called robots.txt in the top-level directory of the website. Notes on the robots.txt Rules: Rules of specificity apply, not inheritance. Always include a blank line between rules. Note also that not all robots […] Continue reading »

Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
WP Themes In Depth: Build and sell awesome WordPress themes.
Thoughts
Finishing up the pro version of Head Meta Data plugin, launch planned this month.
Finally finished my ultimate block list to stop AI bots :) Blocks over 100 AI bots!
After 10 years working late at night, my schedule has changed. I am now a “morning person”, starting my day at 6am or earlier.
Nice update for Wutsearch search engine launchpad. Now with 19 engines including Luxxle AI-powered search.
New version of 8G Firewall (v1.4) now available for download :)
Wishing everyone a prosperous and bright New Year!
I disabled AI in Google search results. It was making me lazy.
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.