Apache Airflow

tools like N8 HEN and Node-RED IDLE between 100-mile and 500-miles of RAM.

Meanwhile, the KESS for Self-Hosting Decision Analysis notes that Apache Airflow versions 2 and 3 require 2.5-gib to 5.0-gib bet idle due to complex component overhead.

Let's start with where Kestra loses.

Deploying the wrong architecture for your problem is a recipe for misery.

Where Kestra absolutely dominates is declarative, container-isolated, multi-language pipelines specified in YAML.

If you have a workflow that needs to run a bash script to download a raw file, pass that file output to a Python container that uses pandas to normalize the data, and then pass that result to a Node.js script to push to a custom API, Kestra is peerless.

Why not just use Apache Airflow for that, though? Airflow is the industry standard data orchestrator.

Everyone talks about Airflow.

Well, we'll go ahead and dive in.

So for today, we're going to talk about how ITTI uses Airflow to orchestrate data across more than 30 companies.

And we'll dig into the custom YAML framework that you've built to make Airflow more accessible to non-technical teams.

Before we get into that, let's start with some background.

So tell me about what ITTI and more broadly what Group of Asquiths does.

Also, we standardize and scale then those solutions to other teams across the organization.

Okay, very cool.

And where does Airflow fit into all of that?

So yeah, basically what we've been trying to think about is the open source ecosystem.

What's the role of that for a Cloud Foundry user? um when a you know cloud foundry nails the 12 factor app and nails all of the developer experience and getting that app to production and so you know what does a developer um what other things does a developer need to make that experience even better and beyond your custom app getting that custom app to production you sometimes need open source dependencies you sometimes need third-party resources that you want to build around so something like uh Apache Kafka or an Airflow or maybe an open source queuing service or maybe middleware or something you want to run that your custom app talks to.

And users have been doing this in various ways.

Developers have been doing this in various ways on CF for years.

One of the things we do is we build everything from scratch.

that's required to run let's say an engine x or to run an apache airflow and we have ways we have built patterns for where we can um you know build only what's necessary on that container and understand all the dependencies without breaking the app so taking away all the all the bloat without actually breaking something that the application needs and then finally it's the helm charts um i don't think a lot of people realize this but bitnami is one of the uh the only vendors that is maintaining helm charts in the industry there's a lot of vendors who are using uh either open source home charts and someone else wrote or they're copying helm charts someone else wrote or they're forking them um but we're an actual we're actually an author of over 140 first-party helm charts that we have built with consistency and security best practices in mind.

So for one, if you're going from an Apache Airflow Helm Chart to an Apache Kafka Helm Chart or to a Neo4j Helm Chart or OpenSearch Helm Chart, you're gonna see consistent sets of parameters that you can control things like logging and monitoring and security settings, which is really helpful for an enterprise who's got to keep track of a lot of balls in the air you don't want surprises every time you go somewhere so we can provide that plus we have the expert we've just built a lot of expertise around you know what's a runtime setting that's insecure what's a privilege that you provide what's like a pod level security privilege that's too that's not safe, that you shouldn't delegate in the Helm chart, or what's a way to store security, sorry, configuration values that have passwords in them that's secure.

Things like that we've built up in our repertoire, and we have patterns that are reusable.

But

also Apache Airflow for pipeline orchestration.

And the Google Cloud platform environment, specifically cloud storage, BigQuery for massive data sets, and even Gemini for AI integration.

And all of this feeds into custom visualization tools like Looker and Streamlit.

Okay, hold on.

If you're listening to this and your eyes are glazing over at the mention of Apache Airflow and BigQuery.

Yeah, it's a lot of jargon.

We need to ground this.

Moving forward, integrations that run without human interaction will require an explicitly configured run as service account.

Meanwhile, orchestration pipelines and managed service for Apache Airflow, which you might remember as Cloud Composer, have reached general availability.

Sensitive data protection content policies are also GA, letting you evaluate content and return allow or block decisions.

For storage admins, Google Cloud Storage batch operations are now much more powerful.

Thank you.

I'm especially excited about today's topic because Airflow is something I worked with very close every day.

Yeah, I'm very excited about this because actually, you know, about today's podcast, we will deep dive into orchestrating strictly second control ETL pipeline.

I think you have like a way of doing things that I think is truly interesting.

Also, how did you survive about the sub-DAG deprecation? Because sub-DAGs have been very popular for a while.

But as you will tell us, you actually did the migration, the migration.

And then also we will talk about adapting custom retry logic all the way through Airflow 3.

So a lot of things to cover.

We alluded to this, but there's so many tools in the data space that are just, they're there, like Spark's there, right? Like sales supports, Spark protocol API, because it had to, right? Because Spark is there and everyone's using it.

And you've got things like, On a data platform, you probably have some orchestration like Apache Airflow, right? Everyone is using it.

And how do you decide which tools you're going to support? And how do you approach that? If I want to call it a problem, I call it a problem of like you're building sale and it's like, well, half my clients, you know, like 80% of the people are using Airflow.

And whether I want them to or not, they run most of their pipelines there.

So I need to be able to... you know, provide an easy operator, like, because data platforms are also not just about the compute itself, but typically in a real-world production environment, right, you've got many complex sort of data pipelines doing all different things, maybe on different systems.

Yeah, great question.

So on our platform, thinking about orchestration tools was really important.

And so one thing that we did from day one was versioning, so job versioning and job runs, which is kind of like the core piece you need if you're going to use like a Prefect or Airflow or Daxter or whatever else.

Then, we will look at how the Gemini Enterprise Agent Platform is evolving with dynamic native user interface capabilities and new enterprise data stores.

After that, we'll turn to data engineering and analytics to discuss major security alerts, database assessments in Cloud SQL, ten billion vector scaling in AlloyDB, and critical updates to both managed service for Apache Spark and managed service for Apache Airflow.

Finally, we'll round things out with some essential networking, security, and identity and access management updates.

Let's jump right in.

And for eligible customers, the standard Emerging Market Edition now has full access to standard AI developer tools.

Over in observability and tracing, Cloud Trace now automatically generates trace spans for tools call operations initiated by remote Model Context Protocol, or MCP, servers.

This includes integrations with Datastream, Policy Troubleshooter, and the managed service for Apache Airflow.

These spans give you an exact step-by-step visual map of what tools your AI agent called and how long each step took, making it incredibly easy to debug agentic behaviors.