What we found on the web about Robots.txt
Robots exclusion standard - Wikipedia, the free encyclopedia
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from ...
en.wikipedia.org
# robots.txt for http://www.wikipedia.org/ and friends # # Please note: There are a lot of pages on this site, and there are # some misbehaved spiders out there that go _way_ too ...
Robots.txt Generator - SEO Tools - Search Engine Optimization, Google ...
Robots.txt Generator - Imposing Restrictions ... Google is making some changes on how automated search results are handled, and it is causing some of our tools to not operate ...
www.google.com
User-agent: * Disallow: /search. Disallow: /groups. Disallow: /images. Disallow: /catalogs. Disallow: /catalogues. Disallow: /news. Disallow: /nwshp. Allow: /news?btcid=
Manual:robots.txt - MediaWiki
robots.txt files are part of the Robots Exclusion Standard. They tell web robots how to index a site. A robots.txt file must be placed in the web root of a domain.
www.cnn.com
Sitemap: http://www.cnn.com/sitemap_index.xml. Sitemap: http://www.cnn.com/sitemap_news.xml. Sitemap: http://www.cnn.com/video_sitemap_index.xml. User-agent: *
www.nytimes.com
# robots.txt, www.nytimes.com 1/21/2009 # User-agent: * Disallow: /adx/bin/ Disallow: /aponline/ Disallow: /archives/ Disallow: /auth/ Disallow: /cnet/
www.w3.org
# robots.txt for http://www.w3.org/ # # $Id: robots.txt,v 1.58 2009/10/30 22:50:57 gerald Exp $ # # For use by search.w3.org. User-agent: W3C-gsa. Disallow: /Out-Of-Date
Entry added by Creative Administrator
Creative Administrator also edited:
Here is what users have to say about Robots.txt

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, sitemaps, a robot inclusion standard for websites.

Welcome to CWAnswers

CWAnswers is your guide to the sprawling world wide web. The directory aims to provide a useful guide made by users. You can share your knowledge as well - simply register and edit your first entry. For questions just contact the team at support - at - cwanswers.com.

Weblinks

Top 10

Things you find nowhere else.

Comments

You must be logged in to post a comment.

No comments yet on this topic. Be the first one!
These recent articles mention Robots.txt
Small & Nimble Trumps Big & Slow In SEO
Searchengineland.com
Giant, slow-moving companies that take over 12 months to implement simple changes to robots.txt files and title tags (true story!) put themselves at a disadvantage to the savvy independent SEOs launching sites every week. The same is true f...