pandas

And speaking from bitter experience, it's a nightmare to work with tabular data.

So anyone who's built a machine learning model with tabular data, you've-- I mean, obviously like Pandas and, you know, scikit-learn has got some stuff in there, but you have to deal with missing values.

You know, what do you do with the categorical features? How do you do some feature engineering? How do you do transformations that reflect the semantics of the problem? You, you, you need to really, really know what you're doing, and it feels kind of kludgy.

You always-- Even though you can make an ML model, you always feel kind of dirty afterwards because you feel like you've, um, done all of these shortcuts and kind of corrupted it in, in some way.

Because if you're expecting a hiring manager or a recruiter, to scroll through or whatever you know go through oh this is the business questions and the source and the queries and all these things before seeing the recommendations and the findings that's just not going to happen so get the buy-in from them by having the findings and recommendations up front in your portfolio project For this deadline, I postpone Python unless your target postings consistently require it.

If they do, substitute some second time, second project time with Kaggle's Python and Pandas lesson.

I actually think Kaggle's lessons aren't bad at all.

So good recommendation there.

Right.

You're not making your own pandas.

You're not making-

Exactly

I started to be able to learn these patterns.

Wes McKinney was the developer of a data manipulation package in Python called Pandas, right? Which is very well known as basically a way within Python to manipulate time series data.

He built it because he was at a QR at a hedge fund and he saw people just crunching time series and spreadsheets and was like,

this is

The agent autonomously unpacked the tabular structures.

It wrote temporary Python scripts in the background using libraries like Pandas, executed them within the sandbox to analyze the headers and detect statistical anomalies, normalized the ledgers, and generated reconciled auto reports.

All without human coding.

And the crucial part here is the data sovereignty.

He has a, a deeper explanation of this and how this might work inside of SQL.

He has an article called Mastering DuckDB when you're used to Pandas or Polars.

So I'll include that link also.

So again, some differences here, and if you're bouncing around and moving between these two, which I think is really common.

So you can do a bunch of tricks to avoid copying memory all over.

So this is especially interesting for data science projects, because if you're using something like Pandas or NumPy, what is a NumPy array? It's literally a C array with some makeup on top.

What is a DuckDB vector? It's literally an array with some makeup on top.

So you can just change the makeup and then you can suddenly access the same data with constant cost, right? You don't have to transform your actual data.

So when you do a query, you may be able to just return a piece of the in-memory chunk instead of going, OK, ours looks like this.

But then we're going to copy a million floats over to this thing in this column and then send it back, right?

so this is what was also like one of the things that uh was one of the realizations that database protocols right like so the way you transfer data from the server to the clients they're actually quite slow uh so this is one of the main frustrations we had seen with the data scientists is not only like oh it's clumsy to set it up and you'd like to start a server and create schemas and whatnot But it's also just to get your data from your NumPy or TensorFlow or Pandas or whatever you're running into the database system and back and forth was super slow.

So you completely remove that boundary.

It's in GeoParquet format, and we only have two layers to start, buildings and highways, and we are doing a weekly update cadence.

So this allows you to quickly query a lot of OpenStreetMap data using tools like DuckDB, QGIS, you can use LawnBoard, Jupyter Notebook, shout out to Dev Seed, Pandas, even ArcGIS Analytics, to quickly do certain queries in the data.

So it just plugs OpenStreetMap data directly into all these tools.

There's a few other solutions out there, and we love them all.

So moving beyond a single calculation, how do we wrangle all that real-world data without just drowning in it?

To handle real scale, you turn to Pandas.

The animal.

The library, actually.

Pandas introduces a data structure called a data frame.

If NumPy is the forklift, Pandas is basically the central nervous system for your data.

Okay, I like

that.