Why ETL and Change Data Capture Break at Mainframe Scale — and What Replaces Them

Yeah, well, and just to build on your example, I think the customer you're referencing had actually been through all of these three.

So it certainly hit home to them.

I mean, something we see a lot of folks do is they'll go down the path of ETL, extract, transform, and load.

And the idea there is that, you know, you have to know the structure of your data and you do something, you know, you use, you know, utility from one of our competitors, maybe you write custom COBOL programs, you know, whatever you want to do, you create, you know, data dependent scripts that can transform your data into something useful.

And of course, the problem with that is that it's very brittle.

Every time your data changes, your scripts have to change.

It can be very slow because ultimately you're going to pass through that data many times.

It wants to transform it, it wants to transmit it someplace, and so on and so on.

And of course, the biggest issue is it's a lot of work.

If you have this site that we're talking about, we're talking about one example file, but you know, as part of an application migration they had going on, there were literally tens of thousands of files.

So, you know, imagine the amount of work if it took you a couple of weeks per file to figure out the structure, do some ETL coding.

I mean, it just doesn't scale that way.

So, you know, that's one path.

But again, it's very common out there.

We see it all the time.

Number two is, And maybe it's because some of the other solutions are so slow.

We tend to see a lot of people trying to not just do one time things, but move that data where it's going to go and then use change data capture to keep it synchronized.

And honestly, the issue here is that because it's so slow to get your data where you want it, once it's there, you don't want to do that again.

You want to just send deltas.

You just want to send the changes across as they occur so that you don't have to absorb that pain over and over again.

But again, the devil's in the details here.

To do that, you have to We're talking about VSAM here.

You have to turn on VSAM logging.

You have to manage all of those logs.

There's performance penalties for doing all of that.