How to Add robots.txt For Wordpress Blog
February 19th, 2009
robots.txt is quite useful for your website when search engine robot(or spider) crawl through your website. It can tell the robot that which content or website you don’t need to crawl. Why do we need to do this for search enginee?
Firstly, for some content in your website maybe it’s unuseful to viewer and never change. such as in wordpress blog the login page. on the other hand, some website maybe duplicated. If search engine crawl much duplicated content in your blog, it is bad to upgrade your page rank. At last, search engine robot crawl less content, the less network bandwidth used by search engine. So all of these reasons make sense to have a robots.txt in your blog.
robots.txt is a TXT file with ASCII coding. After writing it, you should put it in the root of your website server( the same place with sitemap.xml), you can visit with http://www.example.com/robots.txt. For a simple robots.txt there are two rules:
‘user-agent:’ this means the rule is applied for what search engine robots. the ‘*’ means that applying for all ones. This is a list of all existing search engine robots. ‘Disallow’ means which website you don’t want crawl.
My wordpress blog robots.txt is as following:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Allow: /wp-content/uploads/
Sitemap: http://www.starryshen.com/sitemap.xml
The robots.txt apply for all robots. And disallow robots to crawl wordpress system folder such ‘wp-admin’, ‘include’ , plugin and theme folder. Disallow robots to crawl all posts’ comments , feed and trackback. Allow robots to crwal ‘ uploads’ folder because the image what I upload is in this folder.
The last line is your blog’s sitemap file.
6 Comments Add your own
1. Jessicafouff | May 11th, 2009 at 4:35 am
Thanks for posting, definitely going to subscribe! See you on my reader.
2. Arianabene | May 13th, 2009 at 11:33 pm
Wow! Thank you! I always wanted to write in my blog something like that. Can I take part of your post to my site? Of course, I will add backlink?
3. starryshen | May 14th, 2009 at 10:09 am
I don’t see anything in your website. Anyway, you can quote this article in your blog.
4. Ceramic tableware | May 25th, 2009 at 11:08 am
回去试试
5. Car On Water | June 9th, 2009 at 6:01 am
Great post!! I just have to copy it
6. GarykPatton | June 17th, 2009 at 1:32 am
How soon will you update your blog? I’m interested in reading some more information on this issue.
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed