---ROBOTS(Content)---------

If you ever ask an SEO pro, The first thing they will ask you, almost 99% guaranteed, is do you have a robots.txt file? You might ask, what in the blue blazes is a robots.txt file....does it cost money...do I have to hire someone to make it?

The answer in one word: NO! It is probably the easiest thing you will ever do when you are optimizing your site for content, and it is one of the most important things you can do in order to get noticed by search engines. Basically, it is a way of saying to search bots, come look at my pages, but don't look at these pages here...

Just as a side note here...what is a bot you may be asking? In a short answer sort of way, a bot is a program that a search engine sends out to your site to check the site's content and links. It checks the places you link to, your site's organization and content...EVERYTHING! It is a way for search engines to not spend millions hiring people to do the horribly boring job of checking everyone's site out. In the end, you want to do whatever you can to PLEASE THE BOT!!

So how do you do it? Here is the best way I think. Just open up notepad or some other text editor that saves in the .txt format. Then open it up and type the following text:

User-Agent: *

Disallow:

This text will open up your site to all bots that might index your site, this includes all of the different search engines out there. However, you may not want to have your whole site indexed. You may do this in the case of outgoing link pages, search results, or even forum discussions. To stop the bot from indexing a certain area of your site, put the following line and substitute your-page.html with the page you want to keep hidden from bots.

Disallow: /your-page.html

If for some reason, you want to stop all bots from indexing any part of your site, there is also a tag for that. You might do this if you are in construction or changing the site extensively. Note though, it is a very good thing to have your site indexed as much as possible by bots, so try not to block them if you don't have to...

# go away
User-agent: *
Disallow: /

One more thing you may want to do is block the bot from indexing a specific folder (such as private data or data that has nothing to do with your site). In this case, you may use the following tag in addition to the other script you have placed:

Disallow: /your-folder/

It is also important to note that if you change the User-agent script you will select for a certain search engine's bot, such as googlebot. In most cases though, you will just want to leave the asterisk there so that all bots are welcome.

So, in the end, if you have followed these instructions, your robots.txt file may look like the following SEO example:

User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-content/b
Disallow: /wp-content/p
Disallow: /wp-content/themes/askapache/4
Disallow: /wp-content/themes/askapache/c
Disallow: /wp-content/themes/askapache/d
Disallow: /wp-content/themes/askapache/f
Disallow: /wp-content/themes/askapache/h
Disallow: /wp-content/themes/askapache/in
Disallow: /wp-content/themes/askapache/p
Disallow: /wp-content/themes/askapache/s
Disallow: /trackback/
Disallow: /*?*
Disallow: */trackback/

(taken from http://www.askapache.com/seo/wordpress-robotstxt-seo.html)

After all your hard work, DON'T FORGET TO SAVE IT!! Save it as robots.txt and upload it to your site's server! There you have it, a fully-functional, SEO-friendly robots.txt file!

G2