Articles tagged with “msn”
- Unexplained Crawl Behavior Involving Tagged Query Strings
- I need your help! I am losing my mind trying to solve another baffling mystery. For the past three or four months, I have been recording many 404 Errors generated from msnbot, Yahoo-Slurp, and other spider crawls. These errors result from invalid requests for URLs containing query strings such as the following:
http://perishablepress.com/press/page/2/?tag=spam
http://perishablepress.com/press/page/3/?tag=code
http://perishablepress.com/press/page/2/?tag=email
http://perishablepress.com/press/page/2/?tag=xhtml
http://perishablepress.com/press/page/4/?tag=notes
http://perishablepress.com/press/page/2/?tag=flash
http://perishablepress.com/press/page/2/?tag=links
http://perishablepress.com/press/page/3/?tag=theme
http://perishablepress.com/press/page/2/?tag=press
..plus hundreds and hundreds ...
- How to Verify the Four Major Search Engines
- Keeping track of your access and error logs is a critical component of any serious security strategy. Many times, you will see a recorded entry that looks legitimate, such that it may easily be dismissed as genuine Google fare, only to discover upon closer investigation a fraudulent agent. There are many such cloaked or disguised agents crawling around these days, mimicking various search engines to hide beneath the radar. Thus, it ...
- Get Back
- The Internet Archive Wayback Machine is a trip into the online past, offering glimpses of ancient website relics. Reaching back through the virtual dark ages of 1996, the Wayback Machine chronicles over 55 billion pages. Although many of the pages appear incomplete due to missing images, the Wayback Machine provides an invaluable resource, enabling users to experience and learn from the arcane internet of yesterday.
Check out these archaic online offerings:
netscape.com, circa ...
- Robots Notes Plus
- About the Robots Exclusion Standard1:
The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. The information specifying the parts that should not be accessed is specified in a file called robots.txt in the top-level directory of the website.
Notes on the robots.txt Rules:
Rules of specificity apply, not inheritance. Always include a blank line between rules. Note also that not all robots ...