robots.txt for SEO?

I was just doing some research about SEO, search engine optimization, and learned that the lowly robots.txt can help your efforts.

You do have a robots.txt file, correct? That’s the text file that tells the nice robots where they can and can’t go on your site. It’s easy to create in a text editor, that is, not MS Word, but something like TextPad.

Here’s the one I just updated for this site: www.walton.com/robots.txt.

You can use it to disallow search engines, the robots, from indexing files that you specify. I learned that it’s a good idea to disallow images, zip files, pdf files, swf or Flash files, as well as directories that contain your javascript and stylesheets.

Javascript and the Stylesheets (.css files) should be used externally and stored in a separate folder. Use an “include” statement in your web pages to link to them. This will also cut down on the amount of code you have stuffed into that poor web page. Google cares about “code to content” ratio and you could be penalized for too much “stuff” that isn’t content.

You would also put your images and other files in specific folders, just as a housekeeping task. Keep the place organized.

Disallow the robots from indexing those folders. It’ll save time and keep you from possible indexing errors. You want the robots to concentrate on your “money pages” and not on all these extra files.

See how easy and accessible that file is? That’s so they can get at it easily. That also means that your competition and hackers can get at it.

Don’t use it to hide sensitive folders or people will know right where to look for it. Be aware that as much as the “good” robots do as they are told, the “bad” robots do exactly what they are told not to do. Just be careful.

If you tell the robots where to go, they will be happier, your content will be indexed better, and you will have better results on the search engine results pages. As long as the robots don’t rise up and take over the world, we’ll be fine.