Klaus Krippendorff

So what is the actual mathematical equation for doing that?

There are several statistical metrics, like Cohen's kappa or Fleiss' kappa, but the most mathematically robust tool is called Krippendorff's alpha.

Krippendorff's alpha.

Right.

Krippendorff's alpha is a highly sophisticated reliability coefficient.

It calculates the observed variance in the human annotations and subtracts the expected variance that would occur by pure chance.

What makes alpha so powerful is that its math scales perfectly, whether you have two annotators or two hundred, and whether you are dealing with binary categorical labels or continuous numerical variables.

Tea monkeys are never going to randomly guess the exact same specific breed of dog out of a thousand options.

In Appendix C, we report inter-grader agreement across each of our dimensions.

Unsurprisingly, we observe high agreement for the dimensions of model welfare and helpfulness versus harmlessness, but for most others, agreement is weak, Krippendorff's alpha between zero point five and zero point six.

At the same time, each of the graders agree on all six of the headline findings of the paper.

Fourth, the sample sizes we use are small, N less than twenty.