• Stop being a LURKER - join our dealer community and get involved. Sign up and start a conversation.

Reply to thread

We only block the Googlebot user-agent when someone is pretending to be Googlebot.


We use Akamai’s Bot Manager product as part of our configuration to help manage the impact of both trusted & malicious bot traffic to our platform. One of the things this product does is block all attempts to spoof Googlebot. Lots of malicious traffic comes in with a spoofed Googlebot user-agent so it’s in our, and our client’s, best interest to block traffic that pretends to be Googlebot. Since you arent actually a Googlebot [USER=3505]@craigh[/USER], we are blocking you from accessing the site. This is normal behavior for a user/tool that spoofs being Googlebot. All actual Googlebot traffic flows as normal – our reporting shows that Googlebot is serving somewhere in the neighborhood of 10 million webpages per week.