Wordpress Themes - WP Forum at BFA

Wordpress Themes - WP Forum at BFA (http://forum.bytesforall.com/index.php)
-   Header configuration & styling (http://forum.bytesforall.com/forumdisplay.php?f=15)
-   -   robots.txt file is blocking Googlebot (http://forum.bytesforall.com/showthread.php?t=20099)

Ken Driver Apr 11, 2013 02:04 AM

robots.txt file is blocking Googlebot
 
Not sure this belongs in 'header' threads but could not see a more suitable category...
Keep getting an auto-generated email from Google that their googlebot cannot access the site. Google says the issue is the robots.txt file.
hoxtonindustries.com.au/robots.txt
I did not knowingly put this file on the server, nor did the client's SEO consultant, so assume it was auto-generated by either the Atahualpa theme, or by WordPress itself?
Here is the text that WAS in the file:
User-agent: *
Disallow: /_mm/
Disallow: /_notes/
Disallow: /_baks/
Disallow: /MMWIP/

User-agent: googlebot
Disallow: *.csi
The client's SEO consultant has since removed the last two lines, but still I get Google's 'access error' message. How did the file get there? Should I just delete it? What do /MMWIP/ and *.csi mean?

Also, in the Atahualpa Theme Options, there is some Google Analytics code put there by the SEO consultant. Could this code possibly be conflicting with the theme's own SEO Options? Not sure if this is in any way related to the robots.txt file (?) but I can turn off the theme SEO Options if you think that is advisable.

juggledad Apr 11, 2013 03:23 AM

1) Atahualpa does not create a robots.txt file
2) Wordpress does not creates a actual robots.txt file
3) check dashboard->settings->reading->Search Engine Visibility
One of the first steps in debugging any issue is 'turn off all plugins'

Quote:

I can turn off the theme SEO Options
sure, try it ank see what happens


All times are GMT -6. The time now is 09:16 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.