NumPy

sure sure yeah so i mean okay if we're talking about the first angle and the um standardization piece um there was the um yeah the something called the uh the visual effects reference platform was um put together by the visual effects uh organization whatever it was called the alliance or something they put again they put a bunch of people together and made an open source um driven organization and they came up with a virtual uh no it says reference platform um which even covers stuff like the tcc compiler and so on things like python qt are kind of highly featuring in this thing because yeah every Every movie production, every movie software piece needs some kind of UI, right? They don't tend to do stuff about the command line.

So standardizing on things like compilers, on things like Python versions, Qt, libraries for speeding stuff up and doing analysis like NumPy and Boost and things like that.

So, yeah.

Yeah, so it's things like, I mean, okay, obviously the Python thing is going back a long way now, but I think you probably remember from your reddest days where stuff was still written in Python 2, right? And these things have been slow to move out of a lot of organizations, but hopefully...

It's just that these are the tools that rust provides to build those kinds of plugins.

So if you think of tools like Pollers or NumPy or any others that do this kind of concept where some of it's written in a lower level language, IO3 just lets you do that.

I

think there's a lot of low-hanging fruit for people to go apply a profiler to their code and say, you know, it really only matters for these 20 lines of code here.

So you could be like, I tossed an aggregate type error, but I want to see if type error is in there, in that aggregate.

Next up, if you remember NumPy, we've featured it several times now.

NumPy 1.7 is out.

It is officially faster than native NumPy, and not by like 25%.

it is now roughly 1.3 times faster than NumPy over the suite.

So that's pretty cool.

We went from underperforming from native NumPy to now NumPy TS is better.

So I just linked to the changelog.

Okay.

He invented things like SciPy, NumPy, NumFocus.

Anaconda is a company that he built.

He invented the ima- imaging libraries for Python, which is a language that most of modern internet is built on.

In the relational model, a table is an unordered bag of rows, each of which is atomic, meaning that it can't be split up.

Row order isn't defined, although rows may be sorted in a certain manner before being displayed." So to reiterate, Polars DataFrames are a collection of columns, uh, again, kind of coming from the NumPy kind of world, thinking about these individual columns that are being tied together into the DataFrame.

Whereas SQL accesses tables and databases and a table is an unordered bag of rows that are atomic.

So he stresses several areas where this may affect your work.

Did I get zero because my sensor is broken and didn't take any readings, or because it took readings which summed to zero? It's a difference that's easy to work around, but you gotta be aware of it.

And then he talks about another example regarding broadcasting.

Polars follows NumPy style broadcast, whereby if you apply a binary expression with inputs of length N and 1, then the latter 1 gets broadcasted to be the length of N.

He uses this doing centering, a common thing you might wanna do.

So you can do a bunch of tricks to avoid copying memory all over.

So this is especially interesting for data science projects, because if you're using something like Pandas or NumPy, what is a NumPy array? It's literally a C array with some makeup on top.

What is a DuckDB vector? It's literally an array with some makeup on top.

So you can just change the makeup and then you can suddenly access the same data with constant cost, right? You don't have to transform your actual data.

So when you do a query, you may be able to just return a piece of the in-memory chunk instead of going, OK, ours looks like this.

But then we're going to copy a million floats over to this thing in this column and then send it back, right?

so this is what was also like one of the things that uh was one of the realizations that database protocols right like so the way you transfer data from the server to the clients they're actually quite slow uh so this is one of the main frustrations we had seen with the data scientists is not only like oh it's clumsy to set it up and you'd like to start a server and create schemas and whatnot But it's also just to get your data from your NumPy or TensorFlow or Pandas or whatever you're running into the database system and back and forth was super slow.

So you completely remove that boundary.