Agreement Study Example

The FREQ Procedure

Agreement Study Example

Medical researchers are interested in evaluating the efficacy of a new treatment for a skin condition. Dermatologists from participating clinics were trained to conduct the study and to evaluate the condition. After the training, two dermatologists examined patients with the skin condition from a pilot study and rated the same patients. The possible evaluations are terrible, poor, marginal, and clear.

Table 28.2 contains the data.

Table 28.2: Skin Condition Data

	Dermatologist 2
Dermatologist 1	Terrible	Poor	Marginal	Clear
Terrible	10	4	1	0
Poor	5	10	12	2
Marginal	2	4	12	5
Clear	0	2	6	13

The dermatologists' evaluations of the patients are contained in the variables derm1 and derm2; the variable count is the number of patients given a particular pair of ratings. In order to evaluate the agreement of the diagnoses (a possible contribution to measurement error in the study), the kappa coefficient is computed. You specify the AGREE option in the TABLES statement and use the TEST statement to request a test for the null hypothesis that their agreement is purely by chance. You specify the keyword KAPPA to perform this test for the kappa coefficient. The results are shown in Figure 28.6.

   data SkinCondition;
      input derm1 $ derm2 $ count;
      datalines;
   terrible terrible 10
   terrible     poor 4
   terrible marginal 1
   terrible    clear 0
   poor     terrible 5
   poor         poor 10
   poor     marginal 12
   poor        clear 2
   marginal terrible 2
   marginal     poor 4
   marginal marginal 12
   marginal    clear 5
   clear    terrible 0
   clear        poor 2
   clear    marginal 6
   clear       clear 13
   ;
   proc freq data=SkinCondition order=data;
      weight count;
      tables derm1*derm2 / agree noprint;
      test kappa;
   run;

The FREQ Procedure

Statistics for Table of derm1 by derm2

Simple Kappa Coefficient
Kappa	0.3449
ASE	0.0724
95% Lower Conf Limit	0.2030
95% Upper Conf Limit	0.4868

Test of H0: Kappa = 0
ASE under H0	0.0612
Z	5.6366
One-sided Pr > Z	<.0001
Two-sided Pr > \|Z\|	<.0001

Sample Size = 88

Figure 28.6: Agreement Study

The kappa coefficient has the value 0.3349, which indicates slight agreement between the dermatologists, and the hypothesis test confirms that you can reject the null hypothesis of no agreement. This conclusion is further supported by the confidence interval of (0.2030, 0.4868), which suggests that the true kappa is greater than zero. The AGREE option also produces Bowker's test for symmetry and the weighted kappa coefficient, but that output is not shown.

Chapter Contents
Previous
Next
Top