Anyone have experience with data lakes

[douglaskarr]

We've built a platform for our data lake to bring in over a dozen resources. Our CMS event data is in Clickhouse because it's so effective for large streams of event data, that might be overkill for a single dealer. We pull what we need from it into the data lake. We bring in the raw JSON from other APIs and then parse it out into a MySQL database. Each month the market data is refreshed. We've done much of this with me using Claude Code and then my engineering team ensuring it's both secure, stable, and scalable. We're going to be migrating it soon into our production environment.

 

[craigh]

We've built a platform for our data lake to bring in over a dozen resources.

Are you also bringing in DMS data or is this mostly the metadata layer of everything your dealership is producing or working in the market?

 

[douglaskarr]

Are you also bringing in DMS data or is this mostly the metadata layer of everything your dealership is producing or working in the market?

My particular focus is on digital visibility... this is purely dealer CMS, inventory pushed there, and then regional search engine result and listings data.

I will advise one critical thing... while a data lake provides all of your data elements, it does not provide the context that is critical for AI to fully explain it. I'd highly recommend you building out a data schema that explains every data element, how it's updated, benefits, challenges, definitions, etc. I worked for one company that literally built this into a table that was ALWAYS queried with every additional query. This way the data came back with the definitions and AI was able to better contextualize the answers.