Cohen's kappa

Cohen's Kappa is a statistical method for computing inter-annotator agreement. It is generally thought to be a more robust judge than simple percent agreement calculation.

The equation for kappa is:

k(a)= (P(a)-P(e))/(1-P(e)) where P(a) is the true annotator agreement, and P(e) is the probability that agreement is due to chance.

{

}