Directed acyclic graph

Like Pulumi is more programmatic than declarative and Terraform is more declarative.

But at the end of the day, they express a desired state, and there is a translation layer that takes the desired state, like with DAG on the case of Terraform, and applies in the right order, calls the APIs that are affecting the actual resources in the cloud.

It's just a translation, right? So like you said, if I don't have to write to that language, why can't I talk in English and let it make the calls directly? But here is the thing.

Terraform, as an example, or OpenTofu invested heavily in doing the DAG and doing the validation and doing stuff that the agent itself would have to reinvent what they had done to apply things in the right order and give you the rigor that they have already baked into the platform.

It is better, like in the Excel example I've given, to let the agent leverage a tool like Terraform than to learn the underlying API.

It's possible.

Do you think that Terraform is still the primary way your customers are managing their infrastructure? Do you see that as the majority?

I would call it 60% just because of inertia, not because it has a specific advantage.

[laughs]

I thought it was a DAG.

Yeah.

Oh, I guess the Tangle was a DAG.

Yeah.

The Tangle was a DAG.

It was the name of the DAG.

What was that, like distributed graph something graph?

So in like an M structure, if you think about like a DA...

Is your, I think your audience would know about DAGs.

Yeah, yeah.

They definitely know about DAGs.

We taught DAGs.

But we, we have a hard time conveying DAGs to time.

Yeah, yeah, but, but go ahead.

But give it a try.

And we do data quality checks and production reporting and file processing.

We operate almost on thousands of DAGs and run across different schedules and business processes.

So that's how the whole team works on manufacturing data, software systems, and enterprises data systems.

Okay, great.

Wow.

Yeah.

So the DAGs, all the formulas, you know, there's a lot of stuff which happens.

We say like, hey, it's an expense app.

The article announced that Tala, TerraStrux's autolayout algorithm, is now open source under the MPL 2.0 license.

Tala is a new algorithm designed for layouting software architecture diagrams, producing more natural whiteboard-style diagrams compared to traditional DAG-based algorithms.

It balances multiple aesthetic goals like symmetry, flow, and clustering, and supports customization of node positions.

The post showcased comparisons with other layout algorithms, explained TALA's features such as partial node positioning, and discussed its limitations, including longer runtimes for larger diagrams and less effective handling of DAGs.

I think you have like a way of doing things that I think is truly interesting.

Also, how did you survive about the sub-DAG deprecation? Because sub-DAGs have been very popular for a while.

But as you will tell us, you actually did the migration, the migration.

And then also we will talk about adapting custom retry logic all the way through Airflow 3.

We looked at configurations, Doug ran limits, we try to use pools, different dependency setups and so on.

Basically everything that seems like it could help, but every solution solved just one part of the problem that we had.

And at some point we were stuck and not completely desperate, but close because we realized that the problem was bigger than just how we stopped two DAGs from running at the same time.

And the real question was here, how do we control the full execution flow across multiple DAGs, especially when some of them can be triggered at any moment from the application side? And we had to take a step back.

But as soon as you start to interrogate the customer and ask them, why are you doing it? Well, now we have to move away from that.

Well, shucks, I might be talking them out of doing some service here because I'm selling some DAG time.

And that goes to that when we get to that free DAG thing.

That's where this industry is still so wide open on that.

And I'm not sitting here saying it's all wrong and shouldn't be happening because...

Wheat.

He says, uh, Hey Jake, in the process of onboarding with DAG, awesome dude, Beyond Broke Mastermind member.

Glad to have you in the community.

Uh, I'm here from the UK.

Uh, he's the man, Mike, Mike is the man.

Mike's an awesome moderator over there in the mastermind for us.

Uh, he's on the sales team, does a fantastic job at DAG.

If you guys work with our team, you may see or speak with Mike at some point.