If you are trying to guess how you would exclude all bots from some pages, yet allow specific bots to visit even these pages, you need to be careful on the order of the directives in your Robots.txt file.
For example, a robots.txt file containing these lines:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /feed/
would fail to let Mediapartners-Google to index even the /feed/ pages. To give full permission to Mediapartners-Google you need to reverse the order above:
User-agent: *
Disallow: /feed/
User-agent: Mediapartners-Google
Disallow:
You should always put general directions to the top of the robots.txt file, then provide directions for specific bots.
Related Posts:
- Prevent file changes, disable auto run, even disallow connection of USB drives, with USB Write Protect
- Get Word macros for converting tables to csv or txt, adding borders, formatting, extract all tables
- Dim the entire desktop except for the active window, to protect your eyes and gain focus, with Dimmer Pro
- Learn any vocabulary and drill in random or bidirectional order with pfQuizzz
- Encrypt your plain text (TXT) documents, with MyText Notepad
- Punctuation marks may ruin sort order in MS Word
- Exit all or any programs you like with a single click, using Close All Windows
- Save or export to backup or transfer your VLC Hotkeys or all your VLC preferences file
- Some tips for better safety and security on line
- If your PC suddenly became slow, read these tips for giving it some speed
- Some Windows shortcuts you really need
- Simultaneously close some programs and open others with a keyboard shortcut or a single click
