Data lake

I mean, look, a completely different topic.

But I remember when Data Lake were coming out, and all of a sudden, the Data Lake seems a solution for all the problems.

And then you realize, Well, not really.

There are certain things that a data warehouse is actually super efficient at doing and creating a data lake.

In some cases, they ended up creating more data swamp than pure data lake.

But again, it's the pendulum that starts ringing as people kind of like over-rotate.

And I think that probably also in this situation, that ability to right-size, if you think about the sovereignty situation, And not either over-engineer or under-engineer.

Yeah

... uh, just to, to, to comment there And I like to refer to it as more a, a data lake, right? A-

Yeah

... a place where your data securely exists for people to access, right? And I'll just throw out an example as you continue your thought there, like a torque spec.

... uh, have access to that.

And to your point, that collection of customer-specific application data then becomes...

That feeds into your data lake.

That's right.

And so I was listening to this podcast that was from a data analytics vendor CEO, and he was explaining to me, he was explaining to the audience their use of agentic AI.

and you know of course my first thing is like oh go here we go more hype coming in but his his um their use actually made very much made sense to me which was he explained that they're using agentic ai to allow customers to do a natural language query so you just ask the question however you would ask it it it would actually interpret that and help design the query or queries that you needed based upon the data that was stored in the data lake.

And so I went, wow, that's actually really cool.

It's a very narrow, which I think is actually quite appropriate on that usage.

And part of what he was saying was, well, this can be, they've proven even internally, this can be used by non-ITN users to extract data and get answers to things that they want.

But the important part was it actually showed what it was doing so that they could evaluate whether it was approaching it in the right way.

And so I thought that was really important because that ties back into the whole concept of data quality and is the AI hallucinating? Well, you can actually see how it's actually constructing this, what data sources from the data lake it's using and how it's pulling that answer together.

I'm curious if when the user is asking for something, if the AI is then digging in to understand their intent, right? Because it's kind of like stupid questions get stupid answers.

And then healthcare is another one of them.

Let's now talk about some of the terms that will enable us to have a conversation about iceberg data lake.

And in recent years, I'm hearing about data lake houses.

Explain what those are.

Sure.

The way I would think from data lake and data lake house is to take a step back first to a data warehouse.

So essentially, data warehouse is anything which manages your analytical workloads and They were essentially meant to be very structured, schema on right kind of thing and optimized for BI.

And then later came a world of Data Lake with a cloud world wherein you had S3, ADLS.

And it integrates really well with Microsoft's data platforms as well.

So Data Lake and now with, to lose my mind, Fabric, right? Fabric really brings all of the various pieces together.

So with Fabric, you're getting the business intelligence component with Power BI.

You're getting the Azure AI Foundry components.

You're also getting the ability to stand up data platforms like Data Lake or Data Hub, 50 different names to come up with now.

So that's all SQL as well.

I know you said you were a SQL guy.

So I had to go find other options and...

I ended up finding a third party that I was able to connect to the QuickBooks Online APIs, dump content into Data Lake, but then it was a lot of data.

I will say QuickBooks Online, from looking at the data, is not the most intuitive in terms of the databases and how they structure some of the data.

And I very quickly got overwhelmed.

I need to essentially recreate this report.

Help walk me through it.

And it was able to go in and help me generate a semantic model and pull in the different tables from my data lake and my lake house and all of that and really narrow it down.

So at the end of the day, I didn't have 44 tables.

Like, pre, you know, pre-AI, pre-2022 kind of ChatGPT thing, I can't even really re- remember, like, what were we talking about from a tech and, and innovation standpoint? Like, what was the big thing? Was it...

It was like, it was like data lakes, wasn't it?

Mm.

So there was data lakes, and there was kind of understanding the, the big data, right? And so, and you have to remember that fundamentally, we were talking about AI.

It's just LLMs are what people think of when we think of AI, but we were talking about computer vision back then, right? Computer vision was a big thing, and it really had an impact, and still has.

In fact, it's one of the most mature of all of the AIs, right? Is, is computer vision.

We were talking about, yeah, your, your own, you know, BI dashboards not being big data and talking about data lakes.

But Steven, before that, man, like, when I got into this industry, Procore was the new, the new hotness on the block, man.