What AI Can and Can’t Build on Its Own

There was a study by METR of experienced developers using AI.

They thought it made them faster, but it actually made them slower and they didn't really notice.

When your own team says AI is helping, how do you check whether it really is helping them?

Well, I think the study is based upon they reviewed all the check-ins for GitHub and things like that.

And I believe, you know, we're still in the infancy of this.

I think they just turned developers loose and said, hey, go at it.

Here's

AI.

Make yourself more productive with no direction.

And they all went in that direction and just created a ton of iterations, which actually slowed them down.

So since they weren't coding, it felt it was faster, but the data showed they had way more iterations.

And in the end, they were about 20% less productive than they thought they were more.

They thought they were 20% more productive.

I can clearly see how that would happen.

I think you need new type of roles like AI architects to be able to come in and give them direction on how to use the tool as opposed to Just giving them an order.

Make yourself more efficient.

Because I think leadership doesn't fully understand this yet.

And all they want to do is communicate to their C-level leadership that, hey, we're doing AI, right? Because everyone's got to check that box.

But no one is saying what the directive is of the results of that box being checked.