Problems with "CAM" Peer-Review and Accreditation
Stephen Barrett, M.D.
Peer review is a process in which work is reviewed by others who usually have equivalent or superior knowledge. It may be used during the development or execution of a study, as well as afterward. When studies are completed, researchers strive to publish their results in journals so that others can use or criticize the findings and science can advance. Detailed standards for reporting and evaluating studies have been published. The best scientific journals are peer-reviewed by experts; papers submitted for publication are reviewed by two or more expert referees, then accepted, modified, or rejected by the editor. The peer review process is imperfect but can reliably screen out "obviously flawed and unreliable manuscripts." 
Accreditation constitutes public recognition that an educational program meets the administrative, organizational, and financial criteria of a recognized agency. In the United States, educational standards for schools are set by a network of agencies approved by the U.S. Office of Education (USOE) or the Council on Recognition of Postsecondary Accreditation (CORPA). USOE or CORPA do not accredit individual schools, but they approve the national and regional agencies that do so. Almost all such agencies are voluntary and nongovernmental. Accreditation enables credits to be transferable from one school to another and is used as a basis for entering various professions. Accreditation has been a powerful impetus to quality education. But in recent years, the system has been compromised by USOE recognition of agencies that oversee unscientific teachings.
Formalized peer review was a crucial step in the advancement of modern science because it led to the scientific community's culture of sharing and self-criticism. Primary source, peer-reviewed journals of science and medicine began about 300 years ago in France and in England when editors began using "peers" to help determine whether submitted papers would be published . Today, reports from over 5000 peer-reviewed scientific journals are listed in the Index Medicus and its online counterpart Medline. The two most prestigious American medical journals are JAMA (Journal of the American Medical Association) and The New England Journal of Medicine. JAMA has more than 3,000 names in its reviewer-referee file.
The Internet has become a major factor in journal publication. Many printed journals operate Web sites where readers have access either free of charge or on a paying basis. There are also a few exclusively online journals.
Expert Panel Reports
Expert review is also done by scientific organizations and government agencies. Those that have established formal review processes given great weight by the medical community include:
- The Agency for Healthcare Research and Quality (AHRQ), a component of the U.S. Public Health Service, was established in 1989 to enhance the quality, appropriateness, and effectiveness of health services. Formerly called the Agency for Health Care Policy and Research (AHCPR), it has published many clinical practice guidelines in separate versions for clinicians and consumers.
- The Americal Medical Association's Council on Science and Public Health (formerly called the Council on Scientific Affairs) studies many medical issues and reports to the AMA's House of Delegates. Once accepted, these reports help shape AMA public policies and may be published in JAMA.
- The Cochrane Database of Systematic Reviews is an electronic journal produced by the Cochrane Collaboration, an international network of individuals and institutions, established in 1993, that hopes to cover the entire spectrum of medical interventions.
- The National Academy of Sciences issues the Dietary Reference Intakes and many other reports by expert committees.
- The National Guidelines Clearinghouse (NGC) is an Internet-based public resource sponsored by the AHRQ, in partnership with the AMA and theAmerica's Health Insurance Plans. The Web site summarizes more than a thousand clinical practice guidelines that have met its criteria. Most are trustworthy, but the agency that screens them judges the process by which they were developed rather than their contents. As a result, unscientific chiropractic guidelines have slipped in.
- The National Institutes of Health Consensus Development Program, begun in 1977, has held about 100 consensus conferences in which experts meet for several days to discuss a topic and issue a report. Except for its 1997 acupuncture report, which was controlled by acupuncture proponents , its reports have been very valuable.
- The U.S. Preventive Services Task Force publishes recommendations for preventive services that prudent health professionals should offer their patients in the course of routine clinical care. These recommendations represent the pooled judgment of many experts.
Problems with "CAM"
Listing in the Index Medicus is a favorable sign but does not guarantee quality—particularly with information about "complementary and alternative medicine (CAM)." In recent years, several "CAM" journals that mimic the form but lack the substance of good science have been accepted for listing. As a result, Medline searches on "CAM" topics often yield untrustworthy citations. In addition, respectable journals have done a remarkably poor job of screening out low-quality "CAM" manuscripts [4-6]. The British Medical Journal and the Annals of Internal Medicine have both had periods during which they did an especially poor job in keeping out junk "CAM" reports. I believe that most editors are not suspicious enough and most peer reviewers—even for prominent journals—do not know the subject matter well enough to spot the misleading statements, faked data, or improper statistical manipulation used by "CAM" proponents. As a result, physicians everywhere been receiving a steady stream of misleading reports. Unscientific teachings are also percolating through medical schools. Although the AMA Council on Scientific Affairs once urged that "courses offered by medical schools on alternative medicine should present the scientific view of unconventional theories, treatments, and practice as well as the potential therapeutic utility, safety, and efficacy of these modalities" , pressure by proponents and the lure of grant money have led to the creation of courses, departments, and clinics that promote unscientific theories and practices. Most of the grants have come from the National Center for Complementary and Alternative Medicine (NCCAM), which has provided a steady streem of uncritical reports and wasted money sponsoring useless studies . The NCCAM can be appropriately characterized as a cancer that metastasizes misinformation throughout our medical education system.
Similar problems exist with the continuing medical education (CME) system in the United States. Most courses doctors take after graduating from medical school are "regulated" by the Accreditation Council for Continuing Medical Education (ACCME), which provides voluntary accreditation to thousands of CME providers. CME accreditation has great practical importance because (a) many professional groups, hospitals, insurance programs, and licensing agencies (in some states) require CME participation; and (b) accreditation often influences how many people will take the course.
ACCME regulations state that all courses must be based on scientific principles:
- All the recommendations involving clinical medicine in a CME activity must be based on evidence that is accepted within the profession of medicine as adequate justification for their indications and contraindications in the care of patients.
- All scientific research referred to, reported or used in CME in support or justification of a patient care recommendation must conform to the generally accepted standards of experimental design, data collection and analysis.
- Providers are not eligible for ACCME accreditation or reaccreditation if they present activities that promote recommendations, treatment or manners of practicing medicine that are not within the definition of CME, or known to have risks or dangers that outweigh the benefits or known to be ineffective in the treatment of patients .
Unfortunately, ACCME and many of its accredited providers have failed to prevent nonsensical courses from being accredited. (Some quacky groups even hold provider status that enables them to accredit their own programs.) Some quacky groups avoid close scrutiny by ACCME by combining appropriate and inappropriate topics but seeking and getting acccredition for only the appropriate ones. This enables them to look respectable by sayong that the seminar is "approved by ___ for __ hours of continuing education credit" without calling attention to the fact that the rest of the program is not approved. I and a few of my colleagues have complained to ACCME about various organizations and programs that promote quackery. We also met in person with ACCME's executive officer (Murray Kopelow, M.D.) to discuss efficient procedures for preventing quackery-promoting organizations and courses from being approved. ACCME occasionally prevents quacky topics from being accredited. But I have seen no evidence that he is interested in any type of systematic approach that would address the obvious problems.
USOE recognition is supposed to mean that an accrediting agency is "a reliable authority as to the quality of training offered." However, the criteria are primarily organizational. To achieve recognition, the agency must be national or regional in scope and must have appropriate bylaws, procedures, institutional and public representation, "reliability," and autonomy. Individual schools, in turn, must meet criteria set by the recognized agency. The criteria do not include scientific validity. Although much of what is taught in chiropractic, naturopathic, acupuncture, and massage schools is questionable, agencies for each have been recognized. In 2001, an astrology school was accredited.
- Relman AS. Peer review in scientific journals: What good is it? Western Journal of Medicine 153:520-522; 1990.
- Kronick DA. Peer review in 18th-century scientific journalism. Journal of the American Medical Association 263:1321-1322, 1990.
- Sampson W. On the National Institute of Drug Abuse Consensus Conference on Acupuncture. Scientific Review of Alternative Medicine 2(1):54-55, 1998.
- Gorski TN. The Eisenberg data: Flawed and deceptive. Scientific Review of Alternative Medicine Fall/Winter 1999.
- Barrett S. Remote prayer report misrepresented its data. Consumer Health Digest, Nov 19, 2002.
- Sampson W, London W. Analysis of homeopathic treatment of childhood diarrhea. Pediatrics 96:961-964, 1995. (Debunks previously published article)
- Alternative medicine: Report 12 of the AMA Council on Scientific Affairs (A-97), June 1997.
- Sampson WI. Why the National Center for Complementary and Alternative Medicine (NCCAM) should be defunded. Quackwatch, Dec 10, 2002.
- ACCME's Accreditation Policy Compendium, section 2002-B-09, revised Oct 11, 2002.
This article was revised on July 29, 2012.