Data Systems Requirements: What to Define Before You Build

So the topic of today's podcast is actually really important and it's not technical.

So it's a topic of requirements for data systems.

What do you have to know when you want to build a data system? And how do you understand what are the pieces which you have to know to be able to make it correctly and professionally? So yeah, it's not about a tool or a framework.

And obviously, most of us in the data engineering world, we don't create systems from scratch usually.

So we work with systems and architectures which were built before us, because this is how it is.

But sometimes there are opportunities and projects where you have to build something from scratch.

Often, if you've never done it, yeah, you have no idea how to actually do it, what things you have to think about to be able to design such a system.

So there is a whole field of system design.

There are many books on this topic.

But here in this episode, I'm going to explain what are the main pieces of requirements which you have to understand and what are the topics which you have to think about when you design a data system.

And the thing is, some of those steps and pieces are actually kind of obvious if you think about it.

But it can be just helpful to go through all of them.

And sometimes if you've never researched it, it can be a bit confusing in a way.

If somebody asks you, like, what are the questions you have to ask stakeholders or people who are interested in building a data infrastructure? It's just...

Not very obvious what are those questions if you've never researched it.

So here I'm going to cover them and you will be left with a better understanding of what are the requirements and what are the steps you have to go through when you think of designing a data system.

So there are questions you have to think about when you decide how to build a data system.

The first question is what are the sources of a data? Second, what are the consumers? What is the volume of a data? What is the peak load on the system, which is supposed to be? How fast the data should appear in the system? What are the query patterns? What is the availability of a system? So how available should it be? What are the levels of SLA? For how long should we store our data? What is the correctness of data? So how much of a data can be incorrect? Do we have any compliance to worry about? what is the supposed growth of the system, and what is the budget for the system.

This is a general overview, and I will now go through all of those steps and explain them in a little bit more detail.