J Manipulative Physiol Ther. 1993 Jul-Aug; 16(6): 375-83.

United States Department of Veterans Affairs Chiropractic Services Pilot Program evaluation study SDR #86-09: a critique.

Coulter ID.

Research Division, Los Angeles College of Chiropractic, CA.

The following critique of the Chiropractic Services Pilot Program (SDR #86-09) focuses on two major issues: the terms of reference established for the study and the research constraints that arose from either the terms of reference or their interpretation; the technical design and execution of the research. The review suggests that the constraints invalidated the study and ensured that no comparisons are possible between chiropractic and medical care for VA patients based on these results. The constraints resulted in the use of a nonexperimental design, distinct samples being chosen and nonequivalent care settings being compared. The critique also reveals that in each of the design steps (eligibility criteria, sampling, protocols, data collection, analysis, interpretations) there were serious methodological flaws. These ensured that the two populations being compared (chiropractic patients versus medical patients) were in fact noncomparable. In terms of the economic cost comparisons, the design guaranteed the comparison was unfair, pitting a private, fee-for-service chiropractic practice against a not-for-profit, managed-care, federally regulated and budgeted institution. Furthermore, the allocation of costs to the two groups was done inaccurately. The critique concludes that the results are not valid, they cannot be used for generalizing, they cannot be used for statistical analysis and they should not be used to establish policy. The research design and the methodological flaws meant that the objectives of the study could not be met.