Your website might be invisible to ChatGPT and nobody at your store would ever know - AEO / GEO Megathread

[frankis843]

I realized our DI site was blocking ours from ChatGPT too. We sent it to our DI rep and ours is still blocked a week later.
I set up a ChatGPT automation to check the VDP daily. Sometimes it finds info, others days it comes back as blocked.

 

[frankis843]

I heard back from DI today. I'm not totally sure what it means as it is a bit above my skill level.
Has anyone else been able to fix their issue yet?

"Dealer Inspire utilizes best-in-class web security including a web application firewall (WAF), distributed denial of service (DDoS) protection, and bot abuse prevention technology from Cloudflare to protect our customers’ websites. While this technology blocks undesirable bot traffic, Cloudflare also maintains a list of recognized and verified bots that includes benevolent tools like search engine and AI crawlers. You may view the list of verified bots here, but of note, AI engines such as ChatGPT, Google Gemini, and a dozen more are already verified.

While Cloudflare does now offer the ability for websites to block AI crawlers, Dealer Inspire does not have this function enabled as indexing dealer inventory in AI search engines is valuable in this industry. Therefore, our customers do not need to take any action to continue allowing these verified AI crawlers access to their inventory listings.

Both malicious bot traffic and AI crawlers continue to evolve rapidly and having the protection of a tool such as CloudFlare to sort out the good from the bad should bring dealers peace of mind.

The use of Grok may be the core reason why this is being blocked but please let me know if you have further questions or concerns."

 

[TowMichal14]

Most major dealer website vendors still build dynamic VDPs where inventory details, pricing widgets, and package breakdowns only populate client-side via API calls after the initial DOM loads. While traditional Googlebot handles JavaScript execution relatively well these days, the search fetchers for ChatGPT and Claude are heavily optimized for speed and token limits; they frequently bail on heavy client-side scripts and grab whatever raw HTML they can get within a tight timeout.
Another sneaky culprit we run into constantly is overzealous Cloudflare or Akamai bot management rules set at the platform level. The dealership thinks they're just blocking scrapers and lead harvesters, but their WAF is dropping 403 Forbidden on search crawler user-agents before they even hit robots.txt.

 

[joe.pistell]

Most major dealer website vendors still build dynamic VDPs where inventory details, pricing widgets, and package breakdowns only populate client-side via API calls after the initial DOM loads.

Dude, what are you building now?

 

[dealerdata1]

It seems the big 3 have the biggest issues with AI being able to load information off their website due to the VDPs being dynamic

A great idea back in the day but AI really likes having it in easy to read text

The other funny thing too is the AI agents never read the robots.txt or llms,txt so I am almost curious if it is getting scrapped before hand to make note of it, but you never see it in the run logs for the agent unless it is behind a door

I started goofing with implementing a script to catch AI agents that announce themselves when visiting a site and then connects it to an MCP. Kind of like a salesperson opening the front door of your dealership. Not sure how it will work for sales just yet, but service there is awesome opportunity for automated appointment scheduling by AI agents that follow your already internally set rules.

Traffic is still light at the stores I am looking at for AI traffic specific to scheduling service, but it is there.