All the little .txt files you can put in the root directory of your website

♦ Posted by Jeff Starr in Security, SEO, Web Design

Updated June 26, 2024 • 14 comments

The ones I know of:

ads.txt
humans.txt
robots.txt
security.txt

This site makes use of robots.txt and humans.txt. I don’t need ads.txt because 3rd-party ads aren’t currently running on the site, and security.txt seems not necessary as the site’s contact form is easy enough for anyone to find.

I’m guessing there are other “little text files” out there, but I am super busy building a bookstore subdomain for the site with every spare minute. So for now I’ll post the files that I’m aware of; I’m sure in time more will be found and added to the list.

Also: how are people referring to these files? Surely there is something better than “all the little dot-text files you can put in the root directory of your website”.

I think..

Eventually there will be all sorts of these little text files. Anyone can start a text file thing. Like contact.txt for contact information. Or help.txt for support. It could even go beyond site meta and do like friends.txt or any-useful-bit-of-information.txt.

About the Author

Jeff Starr = Web Developer. Book Author. Secretly Important.

14 responses to “All the little .txt files you can put in the root directory of your website”

Jim S Smith 2020/02/20 12:32 pm

I only allow a few text and XML files to be accessible on my server(s) anyway, and disallow any others:

RewriteCond %{REQUEST_URI} .txt
RewriteCond %{REQUEST_URI} !^/(humans|robots|).txt$ [NC]
RewriteRule .* - [R=404,L]

RewriteCond %{REQUEST_URI} .xml
RewriteCond %{REQUEST_URI} !^/(sitemap|search).xml$ [NC]
RewriteRule .* - [R=404,L]

To me, anything else is treated as “probing”. Plus, this puts a stop to one of the methods of plugin-enumeration, like what is used by WPScan.

– Jim S.

Jeff Starr 2020/02/20 1:10 pm • Post Author

Elegant technique for sites with no other .txt or .xml files. Thanks for sharing.

Also about the code:

1) There is an extra pipe | in the 2nd condition, will cause an error.

2) Also, did you include any <IfModule> logic? It looks like WordPress removed them (despite my programmatic efforts to always allow).
- Jim S Smith 2020/02/23 2:10 pm
  
  Huh?
  
  A pipe error?
  
  It may be possible that the “!^/” needs to be changed to: “!(^|/)”, because the “or” pipe should work. I have a bit more complex REGEX in my own site’s htaccess file. This was originally intended to also be usable on a “multi-site, URI-style” site.
  
  I’ll check it further anyway.
  
  – Thanks.
  
  – Jim S.
- Jeff Starr 2020/02/24 1:23 am • Post Author
  
  Yeah you have this:
  
  RewriteCond %{REQUEST_URI} !^/(humans|robots|)\.txt$ [NC]
  
  Look at the pipe symbol after robots.

Rick Beckman 2020/02/20 9:13 pm

Have you heard of placing these files in /.well-known? I’ve seen that with increasing frequency, and apparently it has an RFC. Any idea on how worthwhile separating these files away like this is?

Presumably the root directory URIs for such files could be 301 redirected to the “well known” directory easily enough using .htaccess.

Jeff Starr 2020/02/21 8:37 am • Post Author

I’ve heard of .well-known and redirecting requests for common/related files, but have not heard of placing the files inside of the otherwise hidden .well-known directory. It’s not something I would ever do for any of my own sites, but there is some logic to be found by putting all of the miscellaneous text or whatever files in the same location. I think it might be a bad idea though to make such location a hidden directory, if any of the “well-known” files are meant for human consumption (e.g., ads.txt, humans.txt, security.txt).
Jim S Smith 2020/02/23 2:15 pm

If you use Let’s Encrypt Certificates in your site, and you use the Apache2 (or whatever SAPI) server for your website-owner validation, you may see in your access logs the Let’s Encrypt bot tries to access the validation hash files in /.well-known/acme. So I have had to craft a rather convoluted way to protect this folder AND allow access to only certain files within it.

– Jim S.

Joe 2020/02/21 11:27 am

Jeff you should learn about the .txt file mta-sts and all the email security settings, every website should setup as standard.

p.s. there are 4 major email security things to setup in the dns records of every website.

I won’t name them, instead I leave you to research and find them for yourself.

Jeff Starr 2020/02/21 11:43 am • Post Author

Very interesting, thanks Joe. I found this, looks like it requires a lot of time to read and implement everything. For the “4 major email security things” for DNS, are you talking about SPF, DKIM, and DMARC? What’s the fourth one?
- Joe 2020/02/22 8:16 am
  
  Actually they are all very easy to setup! To confirm the four are: SPF, DKIM, DMARC and MTA-STS.
- Jeff Starr 2020/02/22 10:56 am • Post Author
  
  Right, that makes sense. Will investigate further, thanks Joe.
- Jim S Smith 2020/02/23 2:18 pm
  
  On the subject of DKIM,
  
  Still refining an automated technique. Otherwise, very simple to implement. Far easier than for CSP-headers anyway.
  
  – Jim S.

Paul 2020/04/17 11:51 pm

Maybe add security.txt? Many security researches use it to contact the website owners about a breach.

Jeff Starr 2020/04/18 9:51 am • Post Author

Yes it’s there on the list.

Comments are closed for this post. Something to add? Let me know.

Jim S Smith 2020/02/20 12:32 pm
I only allow a few text and XML files to be accessible on my server(s) anyway, and disallow any others:
```
RewriteCond %{REQUEST_URI} .txt
RewriteCond %{REQUEST_URI} !^/(humans|robots|).txt$ [NC]
RewriteRule .* - [R=404,L]

RewriteCond %{REQUEST_URI} .xml
RewriteCond %{REQUEST_URI} !^/(sitemap|search).xml$ [NC]
RewriteRule .* - [R=404,L]
```
To me, anything else is treated as “probing”. Plus, this puts a stop to one of the methods of plugin-enumeration, like what is used by WPScan.

– Jim S.
- Jeff Starr 2020/02/20 1:10 pm • Post Author
  
  Elegant technique for sites with no other .txt or .xml files. Thanks for sharing.
  
  Also about the code:
  
  1) There is an extra pipe | in the 2nd condition, will cause an error.
  
  2) Also, did you include any <IfModule> logic? It looks like WordPress removed them (despite my programmatic efforts to always allow).
  - Jim S Smith 2020/02/23 2:10 pm
    
    Huh?
    
    A pipe error?
    
    It may be possible that the “!^/” needs to be changed to: “!(^|/)”, because the “or” pipe should work. I have a bit more complex REGEX in my own site’s htaccess file. This was originally intended to also be usable on a “multi-site, URI-style” site.
    
    I’ll check it further anyway.
    
    – Thanks.
    
    – Jim S.
  - Jeff Starr 2020/02/24 1:23 am • Post Author
    
    Yeah you have this:
    
    RewriteCond %{REQUEST_URI} !^/(humans|robots|)\.txt$ [NC]
    
    Look at the pipe symbol after robots.
Rick Beckman 2020/02/20 9:13 pm

Have you heard of placing these files in /.well-known? I’ve seen that with increasing frequency, and apparently it has an RFC. Any idea on how worthwhile separating these files away like this is?

Presumably the root directory URIs for such files could be 301 redirected to the “well known” directory easily enough using .htaccess.
- Jeff Starr 2020/02/21 8:37 am • Post Author
  
  I’ve heard of .well-known and redirecting requests for common/related files, but have not heard of placing the files inside of the otherwise hidden .well-known directory. It’s not something I would ever do for any of my own sites, but there is some logic to be found by putting all of the miscellaneous text or whatever files in the same location. I think it might be a bad idea though to make such location a hidden directory, if any of the “well-known” files are meant for human consumption (e.g., ads.txt, humans.txt, security.txt).
- Jim S Smith 2020/02/23 2:15 pm
  
  If you use Let’s Encrypt Certificates in your site, and you use the Apache2 (or whatever SAPI) server for your website-owner validation, you may see in your access logs the Let’s Encrypt bot tries to access the validation hash files in /.well-known/acme. So I have had to craft a rather convoluted way to protect this folder AND allow access to only certain files within it.
  
  – Jim S.
Joe 2020/02/21 11:27 am

Jeff you should learn about the .txt file mta-sts and all the email security settings, every website should setup as standard.

p.s. there are 4 major email security things to setup in the dns records of every website.

I won’t name them, instead I leave you to research and find them for yourself.
- Jeff Starr 2020/02/21 11:43 am • Post Author
  
  Very interesting, thanks Joe. I found this, looks like it requires a lot of time to read and implement everything. For the “4 major email security things” for DNS, are you talking about SPF, DKIM, and DMARC? What’s the fourth one?
  - Joe 2020/02/22 8:16 am
    
    Actually they are all very easy to setup! To confirm the four are: SPF, DKIM, DMARC and MTA-STS.
  - Jeff Starr 2020/02/22 10:56 am • Post Author
    
    Right, that makes sense. Will investigate further, thanks Joe.
  - Jim S Smith 2020/02/23 2:18 pm
    
    On the subject of DKIM,
    
    Still refining an automated technique. Otherwise, very simple to implement. Far easier than for CSP-headers anyway.
    
    – Jim S.
Paul 2020/04/17 11:51 pm

Maybe add security.txt? Many security researches use it to contact the website owners about a breach.
- Jeff Starr 2020/04/18 9:51 am • Post Author
  
  Yes it’s there on the list.