Cohen's Kappa is a statistical method for computing inter-annotator agreement . It is generally thought to be a more robust judge than simple percent agreement calculation.

The equation for kappa is:

k(a)= (P(a) − P(e))/(1 − P(e))

where P(a) is the true annotator agreement, and P(e) is the probability that agreement is due to chance.