Apache Spark

So in order to create a project, in order to make sure a data science project runs, I needed to do a lot of transformation.

I needed to do a lot of work in Spark and Presto and similar things.

But I'm a bit out of touch with all these modern things.

So, Sahil, it's very nice to have you here so we can talk about all these things that modern data engineers need to use.

There was no good practices around it.

So when we evolved, we now are in the current state, I would say, is interoperable open lake houses.

So if I talk about it, interoperable means that you want your data to be decoupled with the compute right so your storage decoupled from your compute and then what are you also trying to do is you're making it open when you're going through it and data engineering uh i'm pretty sure that's the case with ml i'm not ml guy data engineering is very much motivated by what's happening in open source a lot of things that get that happen in data engineering are inspired by open source let's say for example storage storage is Sparky, which is open source.

Let's say compute.

I knew the identity of at least two people.

So, you know, it was I mean, 2015 was kind of a turbulent year for Databricks because we were having great success with our open source project, Apache Spark.

We were just not having commercial success.

Revenue was, you know, Gap revenue came in at like one and a half million that year.

problem is because, you know, you most likely won't be able to actually unclog or remove this bottleneck.

So you should just turn all your attention to that.

So for me at that time, it was, you know, open source project, check, super successful, you know, tech, Super awesome.

No commercial success.

You can easily go find sample data sets that are in that realm and that range and that size.

The next thing they reach for is typically a commercial tool like Databricks, Snowflake, Dask, or some of these other things that are Spark.

You're distributing the memory of that dataset across many machines or maybe even across one very large machine, but doing it in an attributed manner.

most people probably don't need to go that far.

where they check whether you really know the platform.

That belief is what sends you back to Spark Internals for a third weekend.

Now, if your last stage is still a coding or SQL round, go and study.

That one is an exam.

After that, we will head over to Kubernetes, discussing the release of GKE 1.37 and a container-eyed privilege escalation vulnerability.

Then we will cover image updates in Managed Service for Apache Spark, Cloud SQL's regional endpoints going GA, and big updates in networking, identity, and cloud storage.

We have a lot of ground to cover, so let's get right into it.

First up, let's talk about BigQuery.

This includes CC Insights QA Scorecard, Content Warehouse Synonym Set, DevConnect Account Connector, Discovery Engine Engine, Discovery Engine Serving Config, GKE Hub Fleet, Model Armor Template, Network Security Offsize Policy, Rapid Migration Assessment Collector, Security Center Management Event Threat Detection Custom Module, Storage Insights Dataset Config, and Vector Search Collection.

Config Connector also added new fields like automated backup policy locations for Big Table Table, swap configs for container cluster and container node pools, instance flexibility policies for Dataproc cluster, and organization ref support for network security firewall endpoints 1.1.

Direct reconciliation support has also been added as an opt-in for Gemini Enterprise Agent Platform TensorBoard 1.1. Moving on to data engineering, let's talk about managed service for Apache Spark, which is the new name for Dataproc on Compute Engine and Serverless deployment.

This week brings several new sub-minor cluster image versions across Debian, Rocky Linux, and Ubuntu.

If you wanted to run, like, a big data analytics job, you can also use Hydro.

So it can do things like Spark or Flink would do.

It doesn't care about the performance shape of your events.

They can be many or few, they can be quick or in volume, but it allows you to program a distributed system in a way that, like, is quite general purpose.

Right.

This is where I think a lot of libraries trip actually, and why we've postponed a general purpose solution for all these years is that if you start to answer those questions like you're asking, what about durability? What about replay of dropped messages or things? All these kind of like specific problems typically lead people to specific solutions.

So if you look at Spark or Flink or Kafka or many of these frameworks, they only work in their narrow sweet spot.

They're distributed, but they only work at the stuff they're good at, and they're really bad at other stuff.

Yeah

... or Spark.

Athena.

Yeah.

There is so many, you know, vendors, uh, selling-

Exactly

... and for Spark.

There is good and bad example.

That's where managed services come in.

Providers like Databricks built an entirely managed analytics platform on top of Apache Spark which is open source but complex to configure at scale.

Their pricing model usage-based subscription for compute and storage removes the burden from customers while generating predictable revenue streams.

One downside with SaaS on open source is that if a cloud provider starts offering a competing managed service, they could effectively commoditize what your company built.

This misalignment led to Dockers founder Solomon Hikes stepping aside in twenty eighteen.

The company refocused its strategy around managed services and enterprise features eventually spinning out Mirantis for kubernetes specific tooling From that story we learned that Founders need a clear separation of Revenue generating functions from community driven Ones to keep investors happy without compromising open source ethos.

The decision of whether to go Fully Proprietary Or Keep the Core Open Also hinges on market positioning Some niches Like Security Require strong Compliance Controls that May Be hard to guarantee in a purely os model Take Palo Alto networks They Provide open source components but Their Flagship firewall platform Is entirely closed Their community Contributions Go Into open source libraries for developers but the core Product Remains a Lockin for enterprise Customers in this scenario the Business leverages a Dual approach it Offers An open Codebase to reduce cost Of Entry Then uses proprietary infrastructure to Generate recurring Revenue From Managed Services and Support Contracts conversely there Are Companies that stay Entirely oss Because Their Value Proposition IS Community Collaboration Itself Apache Spark Has Never Offered Paid Support Directly but Instead Relies on Vendor Partnerships Like Databricks or Cloudera those Vendors Monetize By Providing Managed Infrastructure and Specialized Consulting turning the community into a sales funnel rather Than a direct Revenue source The choice of model also affects how you build your distribution.

If your business relies on SaaS, you get immediate visibility of churn.

You don't have to maintain multiple copies, right? And that's the way it's kind of headless.

And when you can bring your own compute, say Spark, or let's say you want to do Databricks, or you want to do, let's say Snowflake, you can connect any of them to that data and start running your, say, analytics workload on top of it.

Okay, I think we're ready now to talk about what is Apache Iceberg.

Where does it fit into this landscape we've been sketching out?

Yeah, there are multiple engine options you have.

You have proprietary ones, the commercial ones, and you have the open ones.

You can use Spark, the Flink, the Trino, which is open source, but you can use Snowflake, you can use Databricks, you can use BigQuery, anything which is compatible to the Iceberg spec can use it.

You also mentioned there are multiple options for catalog.