Publish–subscribe pattern

Now we can do Kafka and have that be able to plug into a system.

Maybe it's like the once the system goes through, it goes through a PubSub and then sends you the nice email confirmation.

We don't have to sort of wait for the transaction to go through.

Those things that we sort of know and loved were, oh, yes, you can do this, but it sort of was required you to do things sort of off platform.

They also connect to a caching mechanism like a key value store.

They might have a PubSub messaging system.

They might use event sourcing and have to deal with backups as well.

And the goal of Stardust is to take all of that and wrap it in one kind of data store, which gives you PubSub messaging and an event bus and a kind of...

It's not a relational database, but we'll get into how it stores data in a minute.

And it has backups integrated as well.

I worked at Publix for a handful of years.

And if anyone knows anything about Publix, PubSubs are the move.

And I remember when I was a kid, bro, PubSub was like, you know, just made you take, they taste a lot different now.

But I remember my dad, he was a Publix manager for 40 years and he would never get a PubSub like every day.

Is that do you know what I'm talking about or?

Is this the PubSubHubbub?

Yes, PubSubHubbub.

Can you explain that a little bit? Because I don't know that that is super incredibly obvious how that works.

I don't even think I know how that works because how is that not polling? I think the main service must be polling or crawling or something, but then it raised an event and notifies people or it...

that was all I was trying to do was put a name to a thing.

And then eventually people kind of associated me with webhooks and were asking for things like a standard.

But even in those, I would always push it off because when you talk about PubSubHubbub, like anything more specific than this general pattern is not as general, you know.

Another interesting design pattern that I came up with for that was when you start thinking about how you want to communicate between the high level control and the hardware, it initially Kind of feels like it's a one-to-one connection, but I felt it would be more powerful if we had a many-to-many connection.

So we used Mosquito and MQTT, PubSub.

Interesting.

And it meant it was really easy, for instance, to just like put a logger onto the system and log every message if you wanted to see what was going on.

Uh, but I think, uh, it, it gives us basically a deadline, right? It doesn't mean that we're screwed.

Like, we've seen, like, a social response actually pretty neutral around MotherDuck because we do have example where, you know, like Confluent was selling Kafka and, um, you know, AWS has been selling, uh, uh, I would say PubSub event for years.

Uh, Google has a Kafka equivalent.

There is so many, you know, vendors, uh, selling-