#562: DuckLake: The Lakehouse That's Just SQL and Parquet

You also mentioned open table formats, and that's going to become relevant to our Duck Lake side of things.

More generally, what are open table formats?

Yeah, so, I mean, they used to be called open table formats, and then I think people are leaning more towards this lake house format name, which I'm not sure why.

I think it's actually maybe because of Databricks, because I think they started calling it the lake house, and then everybody's like, yeah, the

lake house.

Yeah.

But yeah, OpenTable formats are basically, I think Pedro correct me if I'm wrong, but it's basically just some metadata on some parquet files that you can query in a transactional manner, right? And you can do operations also, atomic operations on this data as well.

In the most basic case, right, it's just basically a pointer, like a metadata file with a pointer to a list of files that, you know, all of those files form part of a table, right? And you can also have things like a schema to it, right? Like what data types do you expect from this table? And you can add more complex things like statistics, like, you know, which files contain which data.

And it can get obviously a lot more complex than that, but that's sort of the basics, I think.

Yeah, and I think the interesting thing of the OpenTable formats is, of course, the openness part of it, right? So they have a specification, the file formats where your data is stored, they also have a specification.

So by just reading the specification, anyone should be able to implement a reader and writer to that format, right? So the whole idea of this, or maybe the beauty of this, is that it should be able to solve the problem where you're locked in into a certain system, right? So you suddenly don't need, like, if you start off your database from Oracle, for example, you're kind of locked in.

Like, doing an Oracle migration is known to be painful, to be costly.

So you're going to be using, you're going to be giving Larry Edson a second island in Hawaii in the next few years.

That's fine.

But then if you're actually using open table formats, you kind of solve that problem, right? It's like you can literally switch from different engines and different providers.

And if you're tired of Databricks and now you want to use Noflake or now you want to use your own solution based on DuckDB, you can theoretically quickly jump from these rights and it's also super scalable because if you're just dealing uh with file formats or like a catalog as well but uh your data is in parquet files you can easily distribute your reads on it uh so there's quite some cleverness uh behind it