New Bookstore! Save 20% on books with discount code: LAUNCH
Web Dev + WordPress + Security

All the little .txt files you can put in the root directory of your website

The ones I know of:

ads.txt
humans.txt
robots.txt
security.txt

This site makes use of robots.txt and humans.txt. I don’t need ads.txt because 3rd-party ads aren’t currently running on the site, and security.txt seems not necessary as the site’s contact form is easy enough for anyone to find.

I’m guessing there are other “little text files” out there, but I am super busy building a bookstore subdomain for the site with every spare minute. So for now I’ll post the files that I’m aware of; I’m sure in time more will be found and added to the list.

Also: how are people referring to these files? Surely there is something better than “all the little dot-text files you can put in the root directory of your website”.

I think..

Eventually there will be all sorts of these little text files. Anyone can start a text file thing. Like contact.txt for contact information. Or help.txt for support. It could even go beyond site meta and do like friends.txt or any-useful-bit-of-information.txt.

Jeff Starr
About the Author
Jeff Starr = Designer. Developer. Producer. Writer. Editor. Etc.
USP Pro: Unlimited front-end forms for user-submitted posts and more.

14 responses to “All the little .txt files you can put in the root directory of your website”

  1. I only allow a few text and XML files to be accessible on my server(s) anyway, and disallow any others:

    RewriteCond %{REQUEST_URI} \.txt
    RewriteCond %{REQUEST_URI} !^/(humans|robots|)\.txt$ [NC]
    RewriteRule .* - [R=404,L]
    
    RewriteCond %{REQUEST_URI} \.xml
    RewriteCond %{REQUEST_URI} !^/(sitemap|search)\.xml$ [NC]
    RewriteRule .* - [R=404,L]

    To me, anything else is treated as “probing”. Plus, this puts a stop to one of the methods of plugin-enumeration, like what is used by WPScan.

    – Jim S.

    • Jeff Starr
      Jeff Starr 2020/02/20 1:10 pm

      Elegant technique for sites with no other .txt or .xml files. Thanks for sharing.

      Also about the code:

      1) There is an extra pipe | in the 2nd condition, will cause an error.

      2) Also, did you include any <IfModule> logic? It looks like WordPress removed them (despite my programmatic efforts to always allow).

      • Huh?

        A pipe error?

        It may be possible that the “!^/” needs to be changed to: “!(^|/)”, because the “or” pipe should work. I have a bit more complex REGEX in my own site’s htaccess file. This was originally intended to also be usable on a “multi-site, URI-style” site.

        I’ll check it further anyway.

        – Thanks.

        – Jim S.

      • Jeff Starr
        Jeff Starr 2020/02/24 1:23 am

        Yeah you have this:

        RewriteCond %{REQUEST_URI} !^/(humans|robots|)\.txt$ [NC]

        Look at the pipe symbol after robots.

  2. Rick Beckman 2020/02/20 9:13 pm

    Have you heard of placing these files in /.well-known? I’ve seen that with increasing frequency, and apparently it has an RFC. Any idea on how worthwhile separating these files away like this is?

    Presumably the root directory URIs for such files could be 301 redirected to the “well known” directory easily enough using .htaccess.

    • Jeff Starr
      Jeff Starr 2020/02/21 8:37 am

      I’ve heard of .well-known and redirecting requests for common/related files, but have not heard of placing the files inside of the otherwise hidden .well-known directory. It’s not something I would ever do for any of my own sites, but there is some logic to be found by putting all of the miscellaneous text or whatever files in the same location. I think it might be a bad idea though to make such location a hidden directory, if any of the “well-known” files are meant for human consumption (e.g., ads.txt, humans.txt, security.txt).

    • If you use Let’s Encrypt Certificates in your site, and you use the Apache2 (or whatever SAPI) server for your website-owner validation, you may see in your access logs the Let’s Encrypt bot tries to access the validation hash files in /.well-known/acme. So I have had to craft a rather convoluted way to protect this folder AND allow access to only certain files within it.

      – Jim S.

  3. Jeff you should learn about the .txt file mta-sts and all the email security settings, every website should setup as standard.

    p.s. there are 4 major email security things to setup in the dns records of every website.

    I won’t name them, instead I leave you to research and find them for yourself.

    • Jeff Starr

      Very interesting, thanks Joe. I found this, looks like it requires a lot of time to read and implement everything. For the “4 major email security things” for DNS, are you talking about SPF, DKIM, and DMARC? What’s the fourth one?

      • Actually they are all very easy to setup! To confirm the four are: SPF, DKIM, DMARC and MTA-STS.

      • Jeff Starr

        Right, that makes sense. Will investigate further, thanks Joe.

      • On the subject of DKIM,

        Still refining an automated technique. Otherwise, very simple to implement. Far easier than for CSP-headers anyway.

        – Jim S.

  4. Maybe add security.txt? Many security researches use it to contact the website owners about a breach.

Comments are closed for this post. Something to add? Let me know.
Welcome
Perishable Press is operated by Jeff Starr, a professional web developer and book author with two decades of experience. Here you will find posts about web development, WordPress, security, and more »
WP Themes In Depth: Build and sell awesome WordPress themes.
Thoughts
After 10 years working on my 2010 iMac, my upgrade finally arrived. Shiny new iMac shipped from Ireland :)
Too much caffeine weirds me out. But I love the taste of coffee. So once in a while I enjoy a small cup of decaf. Hits the spot.
Chris Coyier is a truly awesome person. One of the finest people I've ever worked with. Just #gottasayit
Excel won't open CSV file because SYLK format? Open it with text editor and add an apostrophe ' at the beginning of the file, save changes, done.
Displaying too many social media buttons and links all over the place imho makes you look desperate and frankly kinda sad.
Interesting and insightful domain security scanner. Could spend hours playing.
Seeing massive increases in spam and malicious attacks across the board. Nothing out of the ordinary, just a LOT more of it.
Newsletter
Get news, updates, deals & tips via email.
Email kept private. Easy unsubscribe anytime.