Friday, May 3, 2013

How to Custom Robots.txt for Blogspot?

Robots.txt serves to prevent web crawlers work with web robots from accessing all or part of a website if it is publicly viewable. Essentially a robots.txt file on a web site will serve as a request that specified robots ignore specified files or directories when crawling a site.

If you use the wordpress.org platform Blog, robots.txt file can create yourself via file manager in cpanel, but if you are blogspot user then there are steps that need to be before activating robots.txt (disabled by default).

1. Login to your Blogger account and go to your blog dashboard

2. Click on Settings >> Search preferences

3. Edit your Custom robots.txt the choose Yes to enable custom robots.txt content

4. Then fill with the following command

User-agent: Mediapartners-Google
Allow:/
User-agent: Googlebot
Disallow: /?m=1
Disallow: /?m=0
Disallow: /*?m=1
Disallow: /*?m=0
User-agent: *
Disallow: /search?*
User-agent: Twitterbot
Allow:/
User-agent: *
Disallow: /search
Disallow: /p/*
Disallow: /view/*
Allow: /
Sitemap: http://beautifulbeta.blogspot.com/feeds/post/default?orderby=UPDATED


 Note: red letters replace with your own blog

5. Save your jobs

For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but sub.example.com not, the rule which would apply for example.com would not apply to a.example.com

11 comments:

  1. malam gan, blog ini dijual ga?

    klo ya, hubungi ane di email ini ya gan: achoox@gmail.com

    salam,

    ReplyDelete
  2. Hay, áp dụng vào blog liền. Thanks
    http://dulichgiare.abstravel.asia/
    http://dulichonline.info/

    ReplyDelete
  3. by default it is disable means it is not indexed by search engines by changing
    it allows search robots to index all posts. please explain clearly to hotdesishot@gmail.com

    ReplyDelete
  4. your method is good but i think my method is easy to implement check out here chillofyblogging.blogspot.in/2013/06/how-to-setup-custom-robotstxt-in.html

    ReplyDelete
  5. what exactly robot.txt used for ??? i think this not so important to use ???

    Hotel MUrah di Bandung

    ReplyDelete
  6. Hi

    User-agent: *
    Why do you add this line twice?

    Disallow: /p/*
    Why disallowing static pages?

    thanks, hope you answer quick :D



    ReplyDelete
  7. Robots is software of google which crowling any website activity in search engine which do for promoting any website in search engine .
    For more visit : http://www.coolseolist.com

    ReplyDelete
  8. thank you brother. nice knowledge

    ReplyDelete
  9. This is helpfully post for new and advanced blogger.

    http://photos-ofstars.blogspot.com

    ReplyDelete