Philippines staffing research
How reproducible is the monthly report from its source extracts?
Contents
A bounded research protocol for testing whether a recurring report can be rebuilt from its saved definition, extract, and reconciliation record by a reviewer who was not involved in producing it.
Question and scope
This study asks a narrow question: can an independent reviewer reproduce a recurring report's headline totals from the saved definition, the source extract, and the reconciliation record? It is a test of report reproducibility, not a measure of business performance or data quality in general. The unit is one report version for one reporting period. A report that cannot be reproduced is not necessarily wrong, but its reliability cannot be demonstrated from the record. The scope includes the report definition, the extract metadata, the transformation steps, the reconciliation checks, and the final published file. It excludes the subjective judgments that belong to the report owner, such as whether a particular business event should have occurred. Those judgments are recorded as owner decisions and are tested for traceability, not for correctness.
Method and evidence frame
Use a documentary reconstruction method. Select a published report version that has an identified owner, period, and source. Ask a reviewer who did not produce the report to rebuild the headline totals from the saved extract and written definition without consulting the original author. Record every step where the reviewer had to guess, every field that was ambiguous, and every place the saved materials were insufficient. The method separates four layers: the source data, the transformation rule, the reconciliation check, and the published presentation. A failure at any layer is recorded as a reproducibility defect. The reviewer should not silently repair a gap with outside knowledge; the gap itself is the finding. If the reviewer must ask the author a question, record the question and the answer as part of the evidence.
Population and unit of analysis
The population is all recurring reports that support a management, finance, customer, or compliance decision during the observation window. The unit is one report-period-version combination. Reports that share a template but use different source systems are separate units because the reproducibility risk differs. A report with no saved extract may still be tested, but the missing extract is recorded as a limitation rather than treated as a zero. Define the eligible population before sampling. Exclude one-off analyses and drafts that were never published, but do not exclude a report because it looks simple. Simple reports can fail reproducibility when the filter, the period, or the currency is stored only in someone's memory. Record the inclusion rule and the number of eligible reports so the sample has a denominator.
Definitions and measures
The primary measure is the share of sampled report versions whose headline totals an independent reviewer can reproduce exactly from the saved materials. A secondary measure is the share that can be reproduced within a stated tolerance after documented adjustments. Report both counts and denominators. A single percentage without the number of reports and the size of the adjustments is not interpretable. Supporting measures include the number of missing or ambiguous definitions, the number of reconciliation checks that cannot be repeated, the age of the extract, and the time the reviewer needed to reconstruct the report. Record whether the published version matches the saved working version. A report can be reproducible in principle and still fail because the wrong file was published.
Sampling and collection
Use a consecutive or random sample across report families, source systems, and owners. Include at least one report that changed definition during the period and one that depends on a manual extract. Exclude a report only for a documented reason, such as the source system being unavailable for the entire window. Record the exclusion and its effect on the sample. Collect the definition, the extract, the query or transformation note, the reconciliation worksheet, the published file, and the owner's approval. Record the collection date and the version of each artifact. If the source system is live, capture a read-only snapshot or a query result at a declared time. Do not rely on a screenshot that cannot be tied to a period or a filter.
Analysis and reporting
Report the exact-match rate, the tolerance-match rate, and the defect categories. Group defects by missing definition, ambiguous filter, stale extract, undocumented manual adjustment, version mismatch, and calculation error. Show the count in each category and the reports affected. A defect that appears in several reports may point to a template or training problem rather than an individual error. Separate observation from interpretation. A low reproduction rate means the saved materials are insufficient for independent review. It does not prove that the published numbers are wrong. A high rate on a small sample of simple reports does not prove that complex reports are controlled. State the sample size, the report mix, and the review conditions so the reader can judge the strength of the finding.
Limits and sources of error
The reviewer's familiarity with the business can inflate the reproduction rate because outside knowledge fills gaps that a new reviewer would not have. Use a reviewer who is independent of the report but still authorized to see the data. Small samples are sensitive to one difficult report. A report that changed during the period may have two legitimate versions, so the test must specify which version is being reproduced. The method cannot establish that the report supports a correct business decision, that the underlying data is complete, or that the owner's judgment was sound. It tests the traceability of the reporting process. Public control guidance provides general criteria, not a result for a specific sample. Any conclusion should be limited to the reports and period observed.
Operational use
Use the result to decide one of four actions: publish the report as controlled, add the missing definition or extract, narrow the report's claim until the evidence supports it, or pause the report until the owner resolves the gap. Assign an owner and a date to each action. Re-test after the change using the same definitions and a comparable sample so the comparison is meaningful. A reproducible report is not necessarily a useful report, but an unreproducible report cannot be defended when a number is questioned. The practical benefit is a shorter review conversation: the owner can open the saved definition and extract instead of reconstructing the logic from memory. That saves time and makes the report easier to improve.
Review cadence
Repeat the protocol after a system change, a definition change, a major reorganization, or a new reporting owner. A quarterly sample is reasonable for stable reports, while high-risk reports may need a monthly check. Keep the protocol, the sample list, and the defect log together so a later reviewer can repeat the method rather than inventing a new one. Close the review with a named owner, a dated action, and a re-test date. If the report is retired instead of repaired, record why and where the historical version is preserved. A report that disappears without a record creates a new reproducibility gap for anyone who later needs to understand a past decision.