Your website might be invisible to ChatGPT and nobody at your store would ever know - AEO / GEO Megathread

[frankis843]

I realized our DI site was blocking ours from ChatGPT too. We sent it to our DI rep and ours is still blocked a week later.
I set up a ChatGPT automation to check the VDP daily. Sometimes it finds info, others days it comes back as blocked.

 

[frankis843]

I heard back from DI today. I'm not totally sure what it means as it is a bit above my skill level.
Has anyone else been able to fix their issue yet?

"Dealer Inspire utilizes best-in-class web security including a web application firewall (WAF), distributed denial of service (DDoS) protection, and bot abuse prevention technology from Cloudflare to protect our customers’ websites. While this technology blocks undesirable bot traffic, Cloudflare also maintains a list of recognized and verified bots that includes benevolent tools like search engine and AI crawlers. You may view the list of verified bots here, but of note, AI engines such as ChatGPT, Google Gemini, and a dozen more are already verified.

While Cloudflare does now offer the ability for websites to block AI crawlers, Dealer Inspire does not have this function enabled as indexing dealer inventory in AI search engines is valuable in this industry. Therefore, our customers do not need to take any action to continue allowing these verified AI crawlers access to their inventory listings.

Both malicious bot traffic and AI crawlers continue to evolve rapidly and having the protection of a tool such as CloudFlare to sort out the good from the bad should bring dealers peace of mind.

The use of Grok may be the core reason why this is being blocked but please let me know if you have further questions or concerns."

 

[TowMichal14]

Most major dealer website vendors still build dynamic VDPs where inventory details, pricing widgets, and package breakdowns only populate client-side via API calls after the initial DOM loads. While traditional Googlebot handles JavaScript execution relatively well these days, the search fetchers for ChatGPT and Claude are heavily optimized for speed and token limits; they frequently bail on heavy client-side scripts and grab whatever raw HTML they can get within a tight timeout.
Another sneaky culprit we run into constantly is overzealous Cloudflare or Akamai bot management rules set at the platform level. The dealership thinks they're just blocking scrapers and lead harvesters, but their WAF is dropping 403 Forbidden on search crawler user-agents before they even hit robots.txt.

 

[joe.pistell]

Most major dealer website vendors still build dynamic VDPs where inventory details, pricing widgets, and package breakdowns only populate client-side via API calls after the initial DOM loads.

Dude, what are you building now?

 

[dealerdata1]

It seems the big 3 have the biggest issues with AI being able to load information off their website due to the VDPs being dynamic

A great idea back in the day but AI really likes having it in easy to read text

The other funny thing too is the AI agents never read the robots.txt or llms,txt so I am almost curious if it is getting scrapped before hand to make note of it, but you never see it in the run logs for the agent unless it is behind a door

I started goofing with implementing a script to catch AI agents that announce themselves when visiting a site and then connects it to an MCP. Kind of like a salesperson opening the front door of your dealership. Not sure how it will work for sales just yet, but service there is awesome opportunity for automated appointment scheduling by AI agents that follow your already internally set rules.

Traffic is still light at the stores I am looking at for AI traffic specific to scheduling service, but it is there.

 

[Chris Nichols]

Some of the bigger providers could block their LLM crawlers on their CDN - Cloudflare, Akamai, etc. Then you'd never see the block in the robots.txt.

Big companies watch their server costs when making those decisions and don't weigh the downstream impact. Then you might hear them say, "Hey, we have an llm.txt file." But I've never seen evidence that it's effective or moves the needle.

People need education and the tools to be effective for their business. Not some big company making those decisions for you without your knowledge.

 

[nathanp]

Adding one thing from the other side of this. This week I ran 12 buyer questions through ChatGPT, Perplexity and Claude for my own product's site. A quick check on Oct 30 had found us nowhere. Three things I didn't expect:

1. Your test setup can fake a good result. I ran ChatGPT inside a project folder, and it knew I was doing visibility research. One answer named us its top pick. I won't trust that until I rerun it in a temporary chat. Same goes for Claude memory and Perplexity spaces. Test logged out or in a clean session, or you're grading your own homework.
2. Brand-name questions moved first. After I submitted the sitemap to Search Console and Bing, ChatGPT and Perplexity started citing the site when someone asked for us by name. The generic "best tool for X" questions haven't moved yet.
3. Name collisions are real. ChatGPT linked a different company with a similar name as one of our sites. If your store's name is close to another store's, check what the assistants say, then put a plain line on your About page saying who you are and who you are not.

Agree with the robots.txt and JavaScript points above. If the bot can't read the price, it can't quote you.
Anyone else seeing brand-name questions move before the generic ones?
Disclosure: I build Hold the Send, a drafting tool for dealership reps. No link, just sharing what I saw.

 

[Chris Cachor]

I heard back from DI today. I'm not totally sure what it means as it is a bit above my skill level.
Has anyone else been able to fix their issue yet?

"Dealer Inspire utilizes best-in-class web security including a web application firewall (WAF), distributed denial of service (DDoS) protection, and bot abuse prevention technology from Cloudflare to protect our customers’ websites. While this technology blocks undesirable bot traffic, Cloudflare also maintains a list of recognized and verified bots that includes benevolent tools like search engine and AI crawlers. You may view the list of verified bots here, but of note, AI engines such as ChatGPT, Google Gemini, and a dozen more are already verified.

While Cloudflare does now offer the ability for websites to block AI crawlers, Dealer Inspire does not have this function enabled as indexing dealer inventory in AI search engines is valuable in this industry. Therefore, our customers do not need to take any action to continue allowing these verified AI crawlers access to their inventory listings.

Both malicious bot traffic and AI crawlers continue to evolve rapidly and having the protection of a tool such as CloudFlare to sort out the good from the bad should bring dealers peace of mind.

The use of Grok may be the core reason why this is being blocked but please let me know if you have further questions or concerns."

So the central issue is that before AI-agents came along crawler and bot traffic skewed analytics and reporting data because most website vendors weren't savvy enough to distinguish that traffic separately. There's also an added cost to serving that traffic. So aside from the major search engines and ad networks - that traffic was blocked (usually via Cloudflare or Amazon Cloudfront). There's a lot of malicious traffic searching for exploits and things too. So I get it.

But the right way to handle this going forward is two things: allow that bot traffic, but exclude it. Identify that (upstream in the network) as "likely" bot traffic before it even hits your website. Then you can still serve the content (it's cached anyways) but it doesn't mess up reporting data. Then the next step is to support an emerging technology called WebMCP. That basically tells the AI agent here's the AI agent way to consume my website. So essentially the agent can browse your site, add a trade-in, schedule an appointment, etc. and then the agent can report back or even forward you the email or add it to your calendar, etc. Wild times!