Apache Iceberg

But now the question I'm asking is, in the era of AI, do I really need to move data into one comprehensive target, like a consolidated target place? I love the concept of lake house.

I love the fact that Iceberg has really unbundled compute from storage in its true manner, not the cloud's disaggregation, but Any engine can now work on my storage as long as iceberg.

I love that concept.

However, the question that begs is, do I always need to move my data into a lake house? Why can I not leave my data in...

We have a very special guest today, Sahil.

Sahil is a senior technical architect at Snowflake, where he works with enterprise customers running Apache Iceberg lake houses with focus on data engineering and TI-ready platforms.

He's also a maintainer of Schema Charge, an open source database migration project with nearly 500,000 downloads per month.

And he also contributes to Apache OSI, an open source project for synthesizing semantic model exchange.

I'm a senior technical architect here.

I've been here since 2024.

I directly work with customers who run like Iceberg, Lake Houses.

I'm usually in the room where I'm talking mostly about data engineering and its adjacent fields.

So what they're going to do is basically embed DuckDB's vectorized query engine inside of their storage tier.

So basically you can get like on top of S3, like sub second, you know, really cheap query, you know, a query path across, you know, our beloved Parquet files and Apache Iceberg.

So it's a great acquisition.

Of course, you know, the worry is always, wait a minute, isn't that open source? What is going to happen to that project? And yeah, this is very much an acqui-hire situation where they're getting the talents.

That's pretty cool.

Snowflake leapfrogs into the second spot, too.

It's like a massive comeback story for them, honestly, with the work they've done on open sourcing several of their technologies like Polaris and Iceberg, which was already open sourced, but I mean, supporting it and introducing a lot of important technologies.

um movements they with salesforce together releasing the open semantic interchange for instance um google dropped a bit uh not because of their technology although you know as we know those of us that you inhabit this weird world of agents um you know we've been complaining about their models no longer being at the top you know, rung of the echelon here.

So that's a limitation that exists.

And we're also building a system at Fuse that helps with that, which basically is similar to Iceberg.

You just take the data, the very similar specs, you get the metadata of it.

And then in this case, you can just, if the client is asking for the data, you don't need to download the data.