Categories
Troubleshooter Web editing

Robots TXT file: order matters, to disallow all except some bots

If you are trying to guess how you would exclude all bots from some pages, yet allow specific bots to visit even these pages, you need to be careful on the order of the directives in your Robots.txt file. For example, a robots.txt file containing these lines: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /feed/ would […]

Exit mobile version