Elasticsearch

And lastly, fifth is the indexer.

The indexer reads the massive amount of logs and execution histories and indexes them into Elasticsearch or PostgreSQL so they are searchable in your UI.

Where observer, scheduler, executor, worker, indexer, five discrete organs keeping the body alive.

That's a good way to put it.

It is a terrifying report from bleeping computer.

A cybersecurity research collective known as Kinry Labs was scanning the open Internet and stumbled upon a massive, misconfigured Elasticsearch cluster.

Just sitting there.

It was named Paxinfo and it was hosted on IP addresses assigned to Viettel, which is a major telecommunications company in Vietnam.

OK, before we look at the data, I want to understand the failure here.

What exactly is an Elasticsearch cluster and how does a company accidentally leave it exposed on the open Internet?

So Elasticsearch is a highly powerful open source search and analytics engine.

When you have a massive database, say millions of text records, you can't just search it like a simple Excel spreadsheet.

It was later secured after the disclosure, but it's unclear whether the data had already been copied or abused.

Kinryu Labs discovered the Elasticsearch cluster on June 3rd while surveying exposed databases as part of research into ransomware activity.

The cluster, named PaxInfo, contained 29 indices and roughly 107 gig of data.

Its two principal indices held 210 million passenger records and 10 million crew records.

When we were working on this sort of transition off of Elasticsearch, we did a bunch of work to evaluate a different technology, which was reverse indices.

We kept running into small hiccup and small hiccup, but we kept promising the company that our sales team, our post-sales team, who had to really deal with the pain of things slowing down and feeling buggy on the platform, that we were ju- we were just about to solve it.

One more week, one more week, two more weeks.

I was really deep in the query planner and how can-- what's the optimization fences and how to get, like, the table stats to be just right, et cetera, et cetera.

And that kind of kept us alive week by week.

One, one day of optimi- a couple days, two, two days of optimizations would buy us a couple more days at a time while Sam was really taking a new approach at it, which was, "Okay, if we're gonna let you filter on any field on any record- Why don't we use a reverse index over this, right? So what's like the great classic reverse index is Elasticsearch.

And so we took all of this data and we dumped it all in Elasticsearch, and we built a new query layer on top of it to generate these elastic queries and use that to build these very complex reports that our customers needed.

And that, that was really the core and the, the heart of our product to run the automations that I described earlier.

I have been here two and a half years.

So my role here is driving the Elasticsearch AI and platform portfolio at Elastic.

We started out as an open source search engine, and we work on observability, search, security.

I've been here now two and a half years.

There's a many time when you want to find the right bit of information to make the right thing work.

observability, security, and now increasingly a bunch of AI stuff.

Like Elasticsearch has found a lot more users since then, for sure.

Last 15 years have been a blast for the company.

ShinyHunters released the driving record of deceased sex offender Jeffrey Epstein as proof of the hack.

A misconfigured Elasticsearch instance has leaked the personal details of more than 220 million passengers and airline crew members who flew through a Vietnamese airport.

The data is believed to have leaked from an advance passenger information system, a database that stores information on people traveling through airports.

According to Kinriu Labs, the database came from a Vietnamese airport, but they couldn't say which one.

A poorly indexed tag lookup can slow asset rendering and hurt your core web vitals.

Optimize your ORM or use a full-text search engine like Elasticsearch.

On the design side, keep navigation simple.

Users should find what they're looking for in two clicks at most.

Anthropic, Cognition, Notion, Atlassian, Ramp, Superhuman, Linear, Clay, Telus, Grammarly.

It's neat because it's very cost-effective because you're using S3 and, you know, it can still be expensive, don't get me wrong, but it is not like running a Postgres database or Elasticsearch and all of that.

You're using... object storage to store all this stuff, but then make it searchable, which can be whatever I forget.

And they have some maximum documents, unlimited scene, 1 trillion documents at three petabytes.