The kappa or Cohen`s Kappa statistic is a statistical measure of the reliability of inter-consultant variables. In fact, it is almost synonymous with Inter-Rater`s reliability. Cohen`s Kappa formula for two advisors is: where: Po – the relative correspondence observed among the advisors. Pe – the hypothetical probability of a random agreement Once the kappa is calculated, the researcher will probably want to assess the importance of the kappa obtained by calculating the confidence intervals for the kappa obtained. Percentage agreement statistics are a direct measure, not an estimate. There is therefore only a small need for confidence intervals. The Kappa is, however, an estimate of Interrater`s reliability and the confidence intervals are therefore more interesting. Suppose you analyze data for a group of 50 people applying for a grant. Each grant proposal was read by two readers, and each reader said “yes” or “no” to the proposal.
Suppose the data for the tally of disagreements were the following, where A and B are drives, data on the main diagonal of the matrix (a and d) the number of agreements and non-diagonal data (b and c) the number of disagreements: Unfortunately, the limit amounts can estimate or not the amount of the random rate agreement in uncertainty. It is therefore doubtful that the reduction in the estimate of the agreement provided for by the kappa statistics is truly representative of the amount of the coincidence-advice agreement. In theory, the pre (e) is an estimate of the approval rate when advisors advise each position and guess with rates similar to marginal shares, and when the advisors were totally independent (11). None of these hypotheses is justified, so there are wide differences of opinion on the use of Kappa among researchers and statisticians. A final concern about the reliability of advisors was introduced by Jacob Cohen, a leading statistician who, in the 1960s, developed key statistics to measure the reliability of interrater, Cohens Kappa (5). Cohen indicated that there will likely be some degree of match among data collectors if they do not know the correct answer, but if they simply guess. He assumed that a number of conjectures would be speculated and that insurance statistics should be responsible for this fortuitous agreement. He developed Kappa`s statistics as an understanding of this random agree factor. Historically, percentage match (number of chord results/total points) has been used to determine the reliability of Interraters. But a random arrangement on the basis of advice is always a possibility – just as a “correct” answer to a multiple-choice test is possible. Kappa`s statistics take this element into account. 20 disagreements come from Rater B election Yes and Rater A Election No.
15 Differences come from Rater A Yes Election and Rater B vote no. The weighted Kappa allows differences of opinion to be weighted differently and is particularly useful when codes are ordered. :66 Three matrixes are involved, the matrix of observed scores, the matrix of expected values based on random tuning and the weight matrix. The weight dies located on the diagonal (top left to bottom-to-right) are consistent and therefore contain zeroes. Off-diagonal cells contain weights that indicate the severity of this disagreement.