Try it while you read
Open the free Attribute Agreement Analysis tool and click Load sample: it reproduces the case study below.
Theory: measuring the inspector
Visual inspection, leak tests with a go / no-go result or functional checks produce a decision, not a number. A Gage R&R cannot be calculated, but the question is the same: can the system tell good from bad reliably? Attribute agreement analysis answers it on three levels:
- Within appraiser: does each person give the same answer to the same part every time?
- Between appraisers: do all people agree with each other?
- Versus standard: are the answers correct?
Kappa
Percentage agreement is inflated by chance: two people who say “OK” to 90 % of parts agree on most of them by luck. Kappa corrects for this.
κ = 1 is perfect agreement, κ = 0 is agreement no better than chance. Cohen’s kappa compares two raters (an appraiser and the standard); Fleiss’ kappa handles several raters at once.
Key point
Step by step
- Build the sample set: about 30 parts, one third borderline. An expert team decides the correct answer.
- Run the trials: every appraiser judges every part 2 to 3 times, parts in random order and coded.
- Enter the ratings, one row per part and appraiser, plus the Standard rows.
- Read the results and the confusion details: false rejects (good called bad) and false accepts (bad called good).
Case study: visual inspection of painted covers
Three inspectors, Anna, Ben and Carla, judge 30 painted covers three times (the Load sample data).
| Appraiser | Agreement with standard | Kappa | Self-agreement |
|---|---|---|---|
| Anna | 93.3 % | 0.95 | 93.3 % |
| Ben | 86.7 % | 0.91 | 86.7 % |
| Carla | 56.7 % | 0.63 | 56.7 % |
All three appraisers agree with each other on only 43 % of the parts. Carla is inconsistent even with herself, which points to unclear criteria rather than carelessness. The team creates a defect catalogue with photos of acceptable and unacceptable paint runs, adds limit samples at the station, trains all three, and repeats the study with the same parts.
Result
Common mistakes
- Only clearly good or bad parts in the sample.
- Appraisers recognise the parts from earlier trials.
- No standard: agreement between appraisers says nothing about correctness.
Frequently asked questions
What kappa is good enough?
AIAG MSA: kappa above 0.75 indicates good agreement, 0.40 to 0.75 marginal, below 0.40 poor. Agreement with the standard of 90 % or more is commonly required.
Why include borderline parts?
Clearly good and clearly bad parts are judged correctly by everyone. Only borderline parts show whether the acceptance criteria are understood the same way.