Spring Sale! Save 30% on all books w/ code: PLANET24
Web Dev + WordPress + Security

All the little .txt files you can put in the root directory of your website

The ones I know of:

ads.txt
humans.txt
robots.txt
security.txt

This site makes use of robots.txt and humans.txt. I don’t need ads.txt because 3rd-party ads aren’t currently running on the site, and security.txt seems not necessary as the site’s contact form is easy enough for anyone to find.

I’m guessing there are other “little text files” out there, but I am super busy building a bookstore subdomain for the site with every spare minute. So for now I’ll post the files that I’m aware of; I’m sure in time more will be found and added to the list.

Also: how are people referring to these files? Surely there is something better than “all the little dot-text files you can put in the root directory of your website”.

I think..

Eventually there will be all sorts of these little text files. Anyone can start a text file thing. Like contact.txt for contact information. Or help.txt for support. It could even go beyond site meta and do like friends.txt or any-useful-bit-of-information.txt.

About the Author
Jeff Starr = Creative thinker. Passionate about free and open Web.
WP Themes In Depth: Build and sell awesome WordPress themes.

14 responses to “All the little .txt files you can put in the root directory of your website”

  1. Jim S Smith 2020/02/20 12:32 pm

    I only allow a few text and XML files to be accessible on my server(s) anyway, and disallow any others:

    RewriteCond %{REQUEST_URI} .txt
    RewriteCond %{REQUEST_URI} !^/(humans|robots|).txt$ [NC]
    RewriteRule .* - [R=404,L]
    
    RewriteCond %{REQUEST_URI} .xml
    RewriteCond %{REQUEST_URI} !^/(sitemap|search).xml$ [NC]
    RewriteRule .* - [R=404,L]

    To me, anything else is treated as “probing”. Plus, this puts a stop to one of the methods of plugin-enumeration, like what is used by WPScan.

    – Jim S.

    • Jeff Starr 2020/02/20 1:10 pm

      Elegant technique for sites with no other .txt or .xml files. Thanks for sharing.

      Also about the code:

      1) There is an extra pipe | in the 2nd condition, will cause an error.

      2) Also, did you include any <IfModule> logic? It looks like WordPress removed them (despite my programmatic efforts to always allow).

      • Jim S Smith 2020/02/23 2:10 pm

        Huh?

        A pipe error?

        It may be possible that the “!^/” needs to be changed to: “!(^|/)”, because the “or” pipe should work. I have a bit more complex REGEX in my own site’s htaccess file. This was originally intended to also be usable on a “multi-site, URI-style” site.

        I’ll check it further anyway.

        – Thanks.

        – Jim S.

      • Jeff Starr 2020/02/24 1:23 am

        Yeah you have this:

        RewriteCond %{REQUEST_URI} !^/(humans|robots|)\.txt$ [NC]

        Look at the pipe symbol after robots.

  2. Rick Beckman 2020/02/20 9:13 pm

    Have you heard of placing these files in /.well-known? I’ve seen that with increasing frequency, and apparently it has an RFC. Any idea on how worthwhile separating these files away like this is?

    Presumably the root directory URIs for such files could be 301 redirected to the “well known” directory easily enough using .htaccess.

    • Jeff Starr 2020/02/21 8:37 am

      I’ve heard of .well-known and redirecting requests for common/related files, but have not heard of placing the files inside of the otherwise hidden .well-known directory. It’s not something I would ever do for any of my own sites, but there is some logic to be found by putting all of the miscellaneous text or whatever files in the same location. I think it might be a bad idea though to make such location a hidden directory, if any of the “well-known” files are meant for human consumption (e.g., ads.txt, humans.txt, security.txt).

    • Jim S Smith 2020/02/23 2:15 pm

      If you use Let’s Encrypt Certificates in your site, and you use the Apache2 (or whatever SAPI) server for your website-owner validation, you may see in your access logs the Let’s Encrypt bot tries to access the validation hash files in /.well-known/acme. So I have had to craft a rather convoluted way to protect this folder AND allow access to only certain files within it.

      – Jim S.

  3. Jeff you should learn about the .txt file mta-sts and all the email security settings, every website should setup as standard.

    p.s. there are 4 major email security things to setup in the dns records of every website.

    I won’t name them, instead I leave you to research and find them for yourself.

    • Very interesting, thanks Joe. I found this, looks like it requires a lot of time to read and implement everything. For the “4 major email security things” for DNS, are you talking about SPF, DKIM, and DMARC? What’s the fourth one?

      • Actually they are all very easy to setup! To confirm the four are: SPF, DKIM, DMARC and MTA-STS.

      • Right, that makes sense. Will investigate further, thanks Joe.

      • Jim S Smith 2020/02/23 2:18 pm

        On the subject of DKIM,

        Still refining an automated technique. Otherwise, very simple to implement. Far easier than for CSP-headers anyway.

        – Jim S.

  4. Maybe add security.txt? Many security researches use it to contact the website owners about a breach.

Comments are closed for this post. Something to add? Let me know.
Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
GA Pro: Add Google Analytics to WordPress like a pro.
Thoughts
I live right next door to the absolute loudest car in town. And the owner loves to drive it.
8G Firewall now out of beta testing, ready for use on production sites.
It's all about that ad revenue baby.
Note to self: encrypting 500 GB of data on my iMac takes around 8 hours.
Getting back into things after a bit of a break. Currently 7° F outside. Chillz.
2024 is going to make 2020 look like a vacation. Prepare accordingly.
First snow of the year :)
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.