How consistently do reviewers classify offshore article defects?

Research question: when two reviewers inspect the same representative articles under one codebook, how consistently do they classify material defects? Agreement measures application of the written rules; it does not prove that the rules or shared judgments are correct.

Define defect classes with examples and boundary cases before sampling: unsupported claim, excessive scope, duplicated contribution, wrong family, date mismatch, canonical mismatch, broken link, index omission, sitemap omission, and rendering failure. Allow an uncertain category so ambiguity stays visible.

Select a blinded sample containing ordinary pages and known difficult cases without telling reviewers which are expected to fail. Provide the same built commit, sources, manifest, and instructions. Preserve independent first-pass results before discussion.

Report the agreement count for each class alongside the underlying sample size. Where useful, calculate a chance-adjusted measure, but do not hide sparse categories behind one combined statistic. A perfect result from two easy cases says little about a rare high-risk defect.

A Philippines quality specialist may prepare samples, remove unnecessary identifiers, record classifications, and summarize disagreements. The calibration owner decides the final definition and whether published work requires correction. Reviewers should not rewrite evidence after seeing each other’s answers.

Analyze disagreements by cause: unclear definition, missed evidence, different page version, technical access problem, or judgment boundary. The corrective action should match the cause. More training will not fix a codebook that gives two defensible answers.

Check correctness separately with an authoritative source or designated owner where possible. Two reviewers can agree on the same wrong canonical expectation or overlook the same unsupported claim. Agreement is a process signal, not a substitute for ground truth.

Limitations include small samples, reviewer learning, shared training bias, and changing article types. Repeated exposure to seeded cases can also inflate results. Refresh the sample and rerun calibration after material workflow or codebook changes.

Evidence-led conclusion: the study can show which defect definitions reviewers apply reliably and where owner clarification is needed. The useful output is a revised example, narrower rule, or additional verification source—not a broad quality label for the offshore team.