Create a Valid Robots.txt File

Our robots.txt file generator tool creates a robots exclusion file that matches the w3c's robots exclusion HTTP spec. It works for all known search engine spiders and bots that comply with the W3C's HTTP specification and these include all of the major search engines like Google, Yahoo and MSN. Ensure the major search engine spiders like Googlebot, Inktomi Slurp, AlltheWeb's FastCrawler and others index only the content you want them to index. Without a properly formatted robots exclusion file, secure areas of your site may be completely indexed and exposed to the internet by the major search engines. Use our robots.txt generator to exclude secured directories you don't want the major search spiders to access so you can relax knowing you've taken care of one more element in making your web site more search engine friendly.

Your URL:
Allow all robots or none:	All None (to allow all files to be indexed select all)
Not These bots:
And Not These bots:
And Not These bots:
And Not These bots:
Use a "/" symbol to exclude your site from all search engines
Disallow directory:	(example : /clients/)
Disallow directory:	(example : /seo-images/)
Disallow directory:	(example : /seo-members-only/)
Disallow directory:	(example : /seo-secrets/)
Disallow directory:	(example : /seo-help/)

Learn More About Our SEO Firm

Want to learn more about how perform the search engine optimization of customer web sites? Read our seo code of ethics and see if your current seo firm meets up to our strict seo standards. If your prospective seo / sem firm doesn't answer all 20 questions with an emphatic YES, then you need to keep looking. Contact us with any questions you may have regarding search engine placement and first page listings in the major search engines.

Ready to get started? Just fill out the seo custom quote form and we'll have someone contact you immediately.

Create a Valid Robots.txt File

Search Engine Optimization

Self Service SEO

35,540 SEO Tools Users