Methods

Methodology

This material is abstracted with permission from the RAND Corporation.

Background

The RAND/UCLA Appropriateness Method (RAM) was developed in the mid-1980s, as part of the RAND Corporation/University of California Los Angeles (UCLA) Health Services Utilisation Study, primarily as an instrument to enable the measurement of the overuse and underuse of medical and surgical procedures.  An appropriate procedure is one in which "the expected health benefit (e.g., increased life expectancy, relief of pain, reduction in anxiety, improved functional capacity) exceeds the expected negative consequences (e.g., mortality, morbidity, anxiety, pain, time lost from work) by a sufficiently wide margin that the procedure is worth doing, exclusive of cost" (Brook et al., 1986; Park et al., 1986).

The rationale behind the method is that randomised clinical trials—the "gold standard" for evidence-based medicine—often are either not available or cannot provide evidence at a level of detail sufficient to apply to the wide range of patients seen in everyday clinical practice. Although robust scientific evidence about the benefits of many procedures is lacking, physicians must nonetheless make decisions every day about when to apply them. Consequently, it was believed a method was needed that would combine the best available scientific evidence with the collective judgment of experts to yield a statement regarding the appropriateness of performing a procedure at the level of patient-specific symptoms, medical history and test results.

To measure underuse, the RAND/UCLA method was expanded to measure the necessity of clinical procedures (Kahan et al., 1994). Necessity is a more stringent criterion than appropriateness and refers to procedures that must be offered to patients fitting a particular clinical description.  Necessity is more difficult to measure than appropriateness, however, because it involves identifying a group of patients who might have benefited from the procedure, but did not receive it.

Overview of Method

First, a detailed literature review is performed to synthesize the latest available scientific evidence on the procedure to be rated. At the same time, a list of specific clinical scenarios or "indications" is produced in the form of a matrix which categorizes patients who might present for the procedure in question in terms of their symptoms, past medical history and the results of relevant diagnostic tests. These indications are grouped into “chapters” based on the primary presenting symptom leading to a patient’s being referred for treatment or considered for a particular procedure.

A panel of experts is identified, often based on recommendations from the relevant medical societies. The literature review and the list of indications, together with a list of definitions for all terms used in the indications list, are sent to the members of this panel. For each indication, the panel members rate the benefit-to-harm ratio of the procedure on a scale of 1 to 9, where 1 means that the expected harms greatly outweigh the expected benefits, and 9 means that the expected benefits greatly outweigh the expected harms. A middle rating of 5 can mean either that the harms and benefits are about equal or that the rater cannot make the judgment for the patient described in the indication.

The panelists rate each of the indications twice, in a two-round "modified Delphi" process. In the first round, the ratings are made individually at home, with no interaction among panelists. In the second round, the panel members meet for 1-2 days under the leadership of a moderator experienced in using the method. Each panelist receives an individualized document showing the distribution of all the experts’ first round ratings, together with his/her own specific ratings. During the meeting, panelists discuss the ratings, focusing on areas of disagreement, and are given the opportunity to modify the original list of indications and/or definitions, if desired. After discussing each chapter of the list of indications, they re-rate each indication individually. No attempt is made to force the panel to consensus. Instead, the two-round process is designed to sort out whether discrepant ratings are due to real clinical disagreement over the use of the procedure ("real" disagreement) or to fatigue or misunderstanding ("artifactual" disagreement).

Finally, each indication is classified as "appropriate," "uncertain" or "inappropriate" for the procedure under review in accordance with the panelists' median score and the level of disagreement among the panelists. Indications with median scores in the 1-3 range are classified as inappropriate, those in the 4-6 range as uncertain, and those in the 7-9 range as appropriate. However, all indications rated "with disagreement," whatever the median, are classified as uncertain. "Disagreement" here basically means a lack of consensus, either because there is polarization of the group or because judgments are spread over the entire 1 to 9 rating scale.

If necessity criteria are also to be developed, a third round of ratings takes place in which panelists are asked to rate the necessity of those indications that have been classified as appropriate by the panel. The definition of necessity (Kahan et al., 1994a) is that:

• The procedure is appropriate, i.e., the health benefits exceed the risks by a sufficient margin to make it worth doing.
• It would be improper care not to offer the procedure to a patient.
• There is a reasonable chance that the procedure will benefit the patient.
• The magnitude of the expected benefit is not small.

All four of the preceding criteria must be met for a procedure to be considered as necessary for a particular indication. To determine necessity, indications rated appropriate by the panel are presented for a further rating of necessity. This rating is also done on a scale of 1 to 9, where 1 means the procedure is clearly not necessary and 9 means it clearly is necessary. If panelists disagree in their necessity ratings or if the median is less than 7, then the indication is judged as "appropriate but not necessary." Only appropriate indications with a necessity rating of 7 or more without disagreement are judged "necessary."

Figure 1: The RAND/UCLA Appropriateness Method

For more information please see the RAND manual at http://rand.org/pubs/monograph_reports/MR1269/index.html

References

Brook RH, Chassin MR, Fink A, et al. A method for the detailed assessment of the appropriateness of medical technologies. International Journal of Technology Assessment in Health Care. 1986; 2(1): 53-63. 

Kahan JP, Bernstein SJ, Leape LL, et al. Measuring the necessity of medical procedures. Medical Care 1994; 32: 357-365.

Park RE, Fink A, Brook RH, et al. Physician ratings of appropriate indications for six medical and surgical procedures. American Journal of Public Health. 1986; 76 (7): 766-772