GitOps at Enterprise Scale, with Elad Cohen

And once we understand the environment, let's zoom into the original deployment workflow.

Many platform teams tried to reduce friction by giving developers a standard path to production instead of asking every team to build its own process.

How was deployment structured at WSC Sports before the redesign?

Okay, so all of our pipelines were in Azure DevOps and all of our repos.

So we had one template library.

We had a repo called DevOps Pipeline Templates.

We had a Kubernetes template, we had a App Service template, and we had some other ones.

Basically, when you created a pipeline, you had to provide a small set of inputs, such as image name that you want, the environment you're deploying to, requests and limits, and some other configs.

The pipeline did everything in terms of the build, the Docker build, the Docker push, all the environment configurations.

And what it did was help upgrade into the cluster, which means from Azure DevOps, we had multiple service connections for each subscription in the Azure.

which connected to the clusters and did a Helm upgrade every time.

From a developer perspective, it was pretty clean.

You didn't need to understand any internals.

You didn't need to understand the infrastructure layer.

You didn't really need too much.

From the platform perspective, it was centralized and consistent, which means it worked well.

We could scale easily.

Every team could handle their own services and create their own pipelines, which was a win at the time.

And a centralized workflow can be very successful at first because it hides complexity and creates consistency.

But the same abstraction can also hide architectural problems until the platform grows.

So when did you start seeing limits in the existing model?

So as the platform grew, we had more teams, more services, more edge cases.

We needed to add many parameters.

Sometimes, for example, one team needed ephemeral volumes for their pods, which was something that wasn't supported on everything.

So every time we had to keep adding them into the centralized template, and it was much harder to maintain because think of that, you have hundreds of pipelines, you're not really sure what's defined where, and it's getting really hard to track.

And once we understand the environment, let's zoom into the original deployment workflow.

Many platform teams tried to reduce friction by giving developers a standard path to production instead of asking every team to build its own process.

How was deployment structured at WSC Sports before the redesign?

Okay, so all of our pipelines were in Azure DevOps and all of our repos.

So we had one template library.

We had a repo called DevOps Pipeline Templates.

We had a Kubernetes template, we had a App Service template, and we had some other ones.

Basically, when you created a pipeline, you had to provide a small set of inputs, such as image name that you want, the environment you're deploying to, requests and limits, and some other configs.

The pipeline did everything in terms of the build, the Docker build, the Docker push, all the environment configurations.

And what it did was help upgrade into the cluster, which means from Azure DevOps, we had multiple service connections for each subscription in the Azure.

which connected to the clusters and did a Helm upgrade every time.

From a developer perspective, it was pretty clean.

You didn't need to understand any internals.

You didn't need to understand the infrastructure layer.

You didn't really need too much.

From the platform perspective, it was centralized and consistent, which means it worked well.

We could scale easily.

Every team could handle their own services and create their own pipelines, which was a win at the time.

And a centralized workflow can be very successful at first because it hides complexity and creates consistency.

But the same abstraction can also hide architectural problems until the platform grows.

So when did you start seeing limits in the existing model?

So as the platform grew, we had more teams, more services, more edge cases.

We needed to add many parameters.

Sometimes, for example, one team needed ephemeral volumes for their pods, which was something that wasn't supported on everything.

So every time we had to keep adding them into the centralized template, and it was much harder to maintain because think of that, you have hundreds of pipelines, you're not really sure what's defined where, and it's getting really hard to track.