Apache Parquet

JSON events, I just dump them to S3 and then I run Presto on top of that.

Do I already have a lake house or there is more to lake house? Or let's say I take this JSON and convert it to Parquet.

Is it already a lake house or not yet? Like where does a lake house, what is actually a lake house and where does it start?

Okay, sounds good.

Yeah.

So when you ask that question, let's think through it.

What you are describing is a file storage, right? Parquet.

Although Parquet is very rich in metadata, it has...

This is another compression technology that's columnular as well." Um, could we actually just build the entire VertiPaq engine into Park...

against Parquet files? And then we don't need to actually do all the transformations.

You don't need to go and convert your data, like with the tabular VertiPaq engine analysis services.

Analysis services is its own layer, right? So you need to convert data from whatever spreadsheets, CRM systems, whatever that data's coming into, you need to convert, bring it, refresh your semantic model, and bring your data into there.

You could do Direct Query, but you know, that's, that's horrible, right? [laughs] But if you're not...

If you're doing import mode, you need to g- refresh your semantic model, which will take all the data, bring it in compression, and bring it into the semantic model.

But actually, if your data coming into this layer is in delta format already, in, in Parquet, which is just Parquet files with a delta log, then you don't need to do that anymore.

You don't need to go and transform your data because Power BI could read directly against those Parquet files, and actually it still has the same similar level of com- columnular compression, and therefore performance.

And it wasn't until I had to do some training that included some Fabric stuff, and until last year I actually had, you know, my first Fabric thing that a lot of this started to click.

But for anyone out there who's in a similar boat, I would say, like, learn about Parquet files and the whole delta lake kind of thing.

Understand what a lakehouse is, and have a peripheral awareness of some of these pieces.

But I think that'll help you the point that you decide to switch, maybe learn a little bit of Python.

Yes, exactly right.

So, um, I mean, first of all, what is it? It's a library for reading and also writing Apache Parquet files, and I guess we should talk about what Parquet even is.

But yes, um-

A- and, and why you got the idea to do such a thing?

This week brings several new sub-minor cluster image versions across Debian, Rocky Linux, and Ubuntu.

Key updates in these images include Apache Hudi 1.2.0 support in 3.0 images, Parquet footer caching enabled by default for the Lightning Engine using Velux, and Apache Iceberg 1.10 support in 2.2 images.

The Lakehouse catalog also supports autoloading for image versions 2.2 and later.

There is a breaking change to watch out for.

How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first just to learn which Parquet files actually matter.

Duck Lake asks one SQL question instead.

The metadata lives in a real database.

The data stays in plain Parquet.

That's the entire format.

Pedro Holanda joined DuckDB in 2018 when it was still a research prototype at CWI.

They've worked with databases, they've worked with APIs, and they've worked with storage like S3.

But the general idea, I guess maybe the Iceberg original idea was, well, what if we could just use all of S3 for scaling? Think how scalable that is.

Um, and Lens is interesting 'cause basically there is a core extension in DuckDB, uh, for the Lens file format.

There is obviously extensions for Parquet, for Iceberg as well.

So DuckDB, and this is the mechanism that they're describing in this article, DuckDB is the great way actually to combine all of these into one place.

And so you can query both your lake, b- your vector index, and the actual image files, for example, um, to make this work.

As you all may know, DocDB doesn't turn your browser to a database.

The challenge with it is how can I get my data to that, right? And that's where under the hood fuse is doing, is taking the data and converting it to Parquet in here and then passing it to the table.

So that's one way of doing it.

Of course, you may come back and say, well, I have terabyte of data.

This is a perfect example of The bottleneck of this demo that you're seeing right now is the maximum speed that I can get from internet.

It's not the concurrency, and I can set the concurrency as many as I want.

This also allows you to have a smaller chunk when you're building your Parquet file, because that's another piece that you have to keep in mind if you take your parquet file and making it to be a smaller chunk, then you can actually have the data to come faster for the specific partition that you have.

You can align your partition over time, or you can align your partition for space, or combination of work.