Robots TXT file: order matters, to disallow all except some bots

If you are trying to guess how you would exclude all bots from some pages, yet allow specific bots to visit even these pages, you need to be careful on the order of the directives in your Robots.txt file.

For example, a robots.txt file containing these lines

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /feed/

would fail to let Mediapartners-Google to index even the /feed/ pages. To give full permission to Mediapartners-Google you need to reverse the order above:

User-agent: *
Disallow: /feed/

User-agent: Mediapartners-Google
Disallow:

You should always put general directions to the top of the robots.txt file, then provide directions for specific bots.

contact the Editor of TheFreeWindows   Get The Power & Follow!
don't miss   : :   Sleeper   Favorite Launcher   Savy   Gadgetarian
Gadgibility   Power Copy   Windows Rule   SearchALL Gadget   Amazon Gadget