Amazon Redshift

Excel workbooks, comma-separated text files, JSON, XML, PDFs, and Parquet files all live here, which means that messy spreadsheet on your desktop is a perfectly valid starting point.

The database bucket covers the big systems a company runs on, like SQL Server, PostgreSQL, MySQL, Oracle, Snowflake, and Amazon Redshift.

The Azure bucket reaches databases and storage living in Microsoft's cloud, and the Power Platform bucket reaches things like Dataverse, which is the data store sitting behind Power Apps.

Then there's the online services bucket, and that's where it gets interesting.

Yeah.

There's Redshift, which is substantially different from Postgres.

It looks very much like Postgres, but there are a lot of differences.

So there's a commercial dialect for Redshift, if you want to use that.

Cool.

I mean, the fact that you've got this in the first place, so good.

But before we do, you know, we measured seven data warehouse engines.

So Databricks, BigQuery, Redshift, Microsoft Fabric, ClickHouse, Motherduck and Snowflake.

And so, James, as part of the benchmark, you had to get data into all seven of those engines separately.

So, you know, how much of the project's work was it to just land the data in those seven cloud data warehouses?

Yeah, I mean, it's a nonlinear question.

So, you know, there's a few different details that we have, right? So one of the details we have is how big is the data volume that you're working with? So we had, at least for the TPC-DS, we had 1 gig, 10 gig, 100 gig, 1 terabyte, you know, versions of the data set.

So as I change how big my data set is, you know, which engine is good at processing that volume of data and for that engine, what size of compute, you know, so Redshift has, you know, the Redshift processing units or RPUs, Fabric has CUs, compute units, you know, Mother Duck has whatever their sizes are.

Snowflake has, you know, extra small, small, blah, blah, blah.

Like, Datacamp did this, like, back in the early days.

We switched from Redshift to BigQuery, and it was a whole thing that took months.

And that was when Datacamp was, like, a really tiny company.

So I imagine, like, If you're like a multinational company and you've been around for decades, it's going to be a horrible process trying to synchronize everything.

I want to use spot instances when required.

I want to have my reserve instances.

I want to use Redshift when it's the right tool or BigQuery because those are phenomenal tools and we want to be able to use them all.

So at the end of the day, that orchestration would create a new trade.

172.

Understanding how data warehousing on AWS works embark on a journey through the essentials of data warehousing, as well as a practical guide to AWS Redshift implementation.

173.

DREAMFUSION.

This article explains how to apply best software practices to build cool AI products.

59. understanding how data warehousing on AWS works embark on a journey through the essentials of data warehousing, as well as a practical guide to AWS Redshift implementation.

60.

Solid Principles in Java A beginner's GUI DESOLID principles are the principles of objective-oriented programming essential to develop scalable softwares.

Before physical automation, I led the BI platform at Amazon Key.

I separated the Key Redshift environment from the Rings shared cluster, migrated in about six months with only two BI engineers at that time, and no dedicated data engineers.

So with this effort, we managed to cut our dashboard failures by 68%.

And today, this environment supports about 20 plus BI and DE engineers and still is performing at scale.

We have that ability.

So let me dig a little bit deeper into this.

So we also move this data into Redshift for consumption into the various dashboards that are pretty static, but that's another consumer from our system.

So Orkfit, Amazon basically for CloudWatch data to migrate into S3 and Firehose, we can also purge it from CloudWatch as per the retention policy.

a massive history.

And its underlying DNA is so robust that massive tech companies have literally copied its source code and forked it to create enterprise tools like Amazon Redshift and Greenplum.

The defining characteristic of that DNA is extensibility.

The original architects of Berkeley didn't just build a system to hold rows and columns, they built a framework that could be taught to understand entirely new concepts of data.

Yeah.

I mean, basically the gist of this is this is all fitting into Amazon's broader Iceberg strategy, spanning EMR, Glue, and Redshift with both S3 Tables and Glue Data Catalog, also bringing Iceberg rest.

And really this comes down to data s- uh, uh, to data, you know, sustainability as well, because if you store it all in one simple format, and then you can access that format through multiple tools, then you don't have to, you know, do ETL jobs, which both all of us hate.

Um, and so, you know, you save that hassle, and it's one of the big things in sustainability is not copying your data multiple times.