Flaky test

It's one of the last ones in, uh, it was in Kirkland and there was no talk specifically on this topic, but I sat there in the back and I watched every single talk except maybe one part of it.

They talked about what to do with their flaky tests.

And even with AI, we have tests that, I mean, especially with AI, I should say, the general definition I'll use of a flaky test is a test that sometimes it passes and sometimes it fails.

And we are so busy, we don't have a chance to go look at why.

So they need AI assistance.

So in a world where you're building a tool that helps a team create results, A whole crap load, a bare load, I'll call it, of automated tests on a whim.

What's your approach on this concept of flaky tests and making sure if your tool finds a test that fails one out of 10 times, what does that mean?

Yeah.

When I was joining Mozilla, which was early 2010, there was a guy working there, Joel Maher.

It was a project called War on Orange because the flaky tests were called orange tests.

The key thing that we kind of got to the point was that we can't get a green CI.

It's impossible.

When I was at Mozilla, I had friends at Google and Facebook who I'd regularly speak to because some of the open source projects that Mozilla was working on, they were working on it too.

So like there was work on Mercurial, the version control system.

And like we had talked about trying to stick these flaky tests into kind of an area and go like, let's see how often it happens.

Do it all automatically.