Canon Medical Systems USA

Moving Towards Modern Medical Education and Training

Pt. 3C: The challenge to achieving expert performance: reducing the historic 30% error rate.

Anthony Mancuso M.D. - Professor and Chairman of the Department of Radiology - University of Florida - College of Medicine | January 26, 2018

Anthony Mancuso M.D. has a mission to modernize post graduate medical education.  He has  spent the last eight years developing a competency based curriculum and evaluation, based on modern learning theory, with his team at the University of Florida.  They have leveraged extraordinary Learning Technologies to deliver this platform anywhere in the world. In his essay series, Moving Towards Modern Medical Education and Training, Dr. Mancuso will examine in detail: the specific pathway to this adherence to modern learning, educational theory, and the outcome of the application of those principles in this sphere of medical education.

When we review the historic error rates for radiology trainees, we begin to understand the challenge to achieving expert performance.  Waite, et. al. in an excellent recent review article concerning medical errors in diagnostic image interpretation 1 informs us that, “Garland, in groundbreaking work in 1949, reported the error rate in diagnostic imaging interpretation to be 33.3% in positive studies “2. When the denominator is shifted to include all positive and negative studies that might be seen “in a day’s work”, the error number is diluted to approximately 4%, with no accounting for the effect of false positive exams. This initial work by Garland was in the days well before what might be called “complex imaging” were introduced. Garland’s data was related to the interpretation of chest x-rays. Currently, thousands of images per study are generated by way of computed tomography, MRI and ultrasonography creating vast opportunities for error. The cognitive task is enormous. Added to that raw image number cognitive burden, long workdays and high RVU assignments clearly lead to a cognitive overload.
 
That overload is a well- recognized contributing factor to the observed error rates in diagnostic imaging interpretation. In addition, this modern imaging technology, which began in the 1970’s, has an arguably higher medical decision-making and outcome impact than the less complex studies of the pre-“high-tech” imaging era. Finally, this imaging interpretation rate of error is likely equivalent to the error rate in medicine as a whole 1; this is a further indication of a general challenge in medicine that begs to be met by improved graduate, postgraduate and lifelong competency-based education and evaluation.
 
In the following 70 years, the error rate in diagnostic imaging interpretation, proven by many studies to be reasonably accurate at somewhere around 30%, has not declined. 1, 3   Repeated rehashing the debate about what might not or might not constitute an acceptable rate of error will not mitigate the reality of such mistakes. Such enlightened discussion, as that seen to date, only frames a very significant problem. Is the task of reducing this error rate insurmountable? The scope of the problem has not changed for 70 years 1-3 so does this constitute a tacit acceptance of these circumstances? Hardly not. Does this mean that competency in the practice of medicine, as judged by peer equivalency, allows for an error rate of this magnitude to be acceptable? Of course not. Is allowing the persistence of such a rate of mistakes a goal of our graduate and postgraduate medical education and training? Of course it isn’t, and our patients would also not agree with such thinking. It is time for a consistent and deliberate response to reduce this seemingly “sticky” number.

A logical and consistent approach to reduce the rate of mistakes is clearly in order.  At the outset of our work about 8 years ago, at the University of Florida College of Medicine Department of Radiology, we set out to create a competency based evaluation rubric.  We wrote down, in a spreadsheet, what we believe to be the 600 – 800 individual competencies (imaging scenarios) in critical care radiology. We proposed that mastery in these individual conceptual competencies should reasonably define an overall competency in critical care radiology. As a corollary, we considered that proof of true mastery would ultimately create a population of defined experts, engaged in the interpretation of diagnostic imaging studies in the domain of critical care imaging. Such expertise should lead to a significant reduction in the currently unacceptable rate of mistakes in this particular practice domain.
 
Those two posited “givens” are still left to be proven and we have made a start toward that end. After experience over the last 7 years, we also believe instilling the discipline to establish competency in the critical care imaging domain might resonate through the whole of diagnostic imaging practice. The core of our methodology is establishing predictable, disciplined search patterns that contribute to fact-based diagnostic synthesis, as opposed to highly biased synthesis, and that the fact-based discipline will contribute meaningfully to the advancement of accurate medical decision-making. This approach mirrors that of the Weed “coupler” theory discussed earlier in this series of essays 4-8. This discipline, we have observed, when internalized by trainees, results in a general improvement in their work product and professional development outside of the critical care domain.
 
More specifically, since 2010, the UF Department of Radiology has tested about 190 of these individual, conceptual critical care competencies (imaging scenarios) in seven UF Simulations delivered in cooperation with the American College of Radiology (ACR) (Table 3B-1). The ACR became a developmental partner for the delivery of the simulation about 4 years after our own initiation, testing and development of the concept. The Simulation’s purpose was triggered by the proposal of a requirement to fulfill the ACGME Milestone of proving whether the trainees are adequately prepared for the Entrustable Professional Activity (EPA) of independent resident imaging study interpretation; this allows for remote attending radiologist supervision during after-hours practice. This EPA goal was then scheduled for implementation in 2018 9. Our intent was to have a generally available objective, reproducible and reliable tool to be responsive to this goal in place well before that go live date. The tested individual competencies have been distributed as follows:
 
 

Crossover competencies in these organ systems include instances seen in the pediatric population, as well as other crossover topics such as those inherent in vascular and trauma scenarios. Over 500 residents in over 30 programs throughout the United States have participated in these Simulations. The size of individual programs runs anywhere from about 5 to 12 residents per year with a reasonable mix of smaller, medium, and larger programs included.
 
Each Simulation includes 65 cases to be completed over an eight hour remotely (attending not in house) supervised “shift” experience. (Figure 3B-2) In each of the 65 cases, all of the DICOM images for the case are submitted and interpreted using a complete suite of workstation diagnostic tools including full multiplanar and 3D capabilities; that functionality currently made possible by the Visage Corporation. The trainees determine whether the study is normal or abnormal. If abnormal, the trainees must type into a simulated online consultation form exactly what they believe to be the essence of the information the study contributes to medical decision-making and what they would communicate in that regard to the referring provider. Beyond that, they are asked to type into the online form whatever suggestions for further evaluation or additional insights they might provide that would be specifically relevant to the case at hand.

 
Figure 3B-2.  Components of competency for critical care imaging interpretation and communication
 
More specifically in a posited abnormal study, this interpretive exercise must include a suggested diagnosis or differential diagnosis, and if appropriate likely next imaging steps as well as establishing the level of required communication depending on the acuity of each particular scenario. In other words, the trainee is documenting consulting and reporting activity, as it occurs routinely, by an integrated part of the care team. To date almost 50,000, in aggregate, such written responses to individual diagnostic imaging scenarios have been graded and analyzed. The overall accuracy rate of this self-discovery Simulation exercise for residents in the second half of their R2 year is 66% with a range of about 48% to 84%. The mean accuracy rate value just given produces a 34% error rate which bears an uncanny similarity to the 33.3% error rate first revealed by Garland in 1949 11 and consistently confirmed since 1949.10,12
 
In 2018, it will be required that radiology programs trainees demonstrate readiness for independent interpretation of imaging studies and that readiness be objectively documented 9. However, the Milestones documents to date do not specify acceptable true competency evaluation rubrics for such a proposed “objective” documentation. The ACGME milestones “Envelope of Expectations” embody the concept of demonstrable competency (Figure 3B-1). Those ACGME expectations assume, based on the following published information, such competency/ readiness will be reached toward the end of the second year of training or sometime early during the third year and, before the “Entrustable Professional Activity” of shift coverage with long-distance supervision is allowed.
 
Summary analysis of the simulation data in the following graph (Figure 3B-3), for programs who have tested residents in all 4 years, illustrates the mean test score range as a surrogate for accuracy in consulting and reporting performance:
 


 
Figure 3B-3. Performance by resident year (R1=first year of radiology) for programs testing residents in all 4 years. The average case score with 95% confidence intervals (A) as well as the percent of cases with score greater than 3 and greater than 7 (B) are shown.
 
These data suggest a plateau (Figure 3B-3) that might be reasonably expected to persist beyond training without a true ongoing commitment and opportunity for “Deliberate Practice”. Therefore, we must consider whether this flattening performance level indicates that postgraduate training programs accept such an error rate in defining competency of their trainees as they embark upon the rest of their careers. Such consideration seems to be reasonable since error rates have not really improved given the current methodologies employed in continuing medical education and the evaluation rubrics employed to “validate” ongoing sustained improvement in competence toward expert. These persistent error rates, across all 30 programs participating in the simulations, can be viewed as “achievement gaps” or “education opportunity gaps”. We prefer to view them as educational opportunity gaps, in part created by the haphazard nature of our curriculum in diagnostic radiology training, the lack of a completely defined curriculum and deficiency in our observational training methodology.
 
We have also analyzed the root cause of errors in the just over 27K of 43.5K responses to the Simulation. There is a roughly fourfold odds of observational errors (~75%) over interpretive errors (~20%) with the remaining 5% consisting of combined observational and interpretive errors. 13 The radiology literature contains many articles about factors contributing to observational and interpretive errors. It is useful to think about those causes in trying to eliminate them as contributing factors. However, radiology educators must use the summary data just presented to consider that we may not be truly teaching our trainees to gather the entire factual basis for calling a study positive or negative, with a high degree of confidence. Also, our current methods may not be effective in applying those observations/facts in a particular clinical context. Please recall the definitions of competency earlier discussed earlier in this essay as expressed by Dreyfus for Stage 1:”Novice: Rule-based behavior, strongly limited and inflexible”. It is at this stage that we must exploit the most appropriate adult learning skills to lay down the fundamental basis of diagnostic imaging interpretation and related decision-making, that being logical and reliable, and perhaps scenario driven, observational discipline. This discipline is currently subject to haphazard educational experiences across the range of our residency programs. Lack of observational discipline clearly leads to error.
 
New approaches in our educational methodology, to these tenacious shortcomings, might begin to improve the situation. Based on our simulation experience new approaches are necessary so that our trainees can reduce the rate of both types of errors. There should be an initial, intense focus on the elimination of observational errors since proper observations are the factual basis for determining whether study is positive or negative and form the core knowledge for a proper and useful thought synthesis and related reporting when a study is positive.
 
These gaps whether they are of the achievement or educational opportunity variety can cause very significant harm. These proven error rates in training might suggest peer equivalency and, therefore, competency at the end of training (as competency seems to be currently defined) allow for a general error of about 20-25%. This is slightly better than the traditionally cited mistake rate of one out of three on positive studies 17-19  Further, careful analysis of the individual competencies error rate in this simulation experience presents a different picture. Table 3B-2 shows specific error rates from 10 selected simulated competency concepts calculated from the pooled simulation data comprising just under 44K total responses.

 
Table 3B-2. Error types for 10 representative case concepts

*”Passing a question” requires 6 out of a possible 10 points per case. Therefore, % below 4 points out of 10 clearly identifies an educational opportunity or achievement gap for that scenario- observational/interpretive or both
 
Figure 3B-4. Error types for 10 representative case concepts (same data at Table 3B-2). Blue=Observational, Red=Both, Orange=Interpretative. The horizontal axis scale is set such that the right edge of the orange bar falls on the percentage of all cases with score <4.
 
These are a relatively small subset of specific competencies that showed error rates 2 to 3 times greater than 25 or 30%. These results tell a different story with regard to potential harm than do the aggregate mistake rate figures. It is generally held that while the error rate may be as high as about 30% in positive diagnostic studies that about 5% of those errors have the capacity to cause significant harm in the general practice of radiology. It appears that that estimate does not take into account specific competency (scenario) error rates.  These individual error rates should not persist in the high-stakes practice of critical care radiology. They must be systematically discovered and eliminated both during the training and, considering the potential plateau we observe, beyond completion of training. The discovery of areas in need of targeted improvement in training methodology requires a sufficient competency-based evaluation rubric. Curing the educational opportunity gap requires a new look at how we educate.  The ultimate goal would be to markedly reduce the current rate of error that likely remains unchanged for 70 years as suggested in a fairly large body of the radiology literature and which is further reflected in the limited reference material provided here 1-3.
 
In radiology education the root problem with our “teaching to the test”, and the corollary, residents spending a disproportionate amount of time “studying to the test” is that there is no curriculum. Our diagnostic imaging educational system relies on very important but haphazard exposure to curriculum during the readout sessions with attending radiologists mainly during regular hours. This is the best part of the current educational environment. Otherwise the transfer of knowledge is by often sporadically attended case conferences often delivered by trainees themselves (though with faculty input), standard textbook assignments at the discretion of individual programs as well as, lectures and studying of teaching files (also at the discretion of individual programs) with varying levels of access to such resources. This traditional and inadequate approach to diagnostic imaging education does not lend itself to covering a specific curriculum that can be proven to be mastered.
 
What then happens to the process of trainee evaluation? The testing rubric defaults also to traditional methods such as multiple choice questions with a few selected images. Such an evaluation in no way reflects what an interpreter of diagnostic imaging studies must do, in real life situations, in order to meaningfully contribute to medical decision-making. The trainees then, without a defined curriculum and with a background in their prior educational system of memorization and regurgitation of facts, default to discovering the likely test content and focusing on “gaming of the test” rather than what really makes a difference to patients, their true clinical competency. This is entirely understandable. Most of the trainees, who through their entire educational experience from primary and secondary school through and including graduate medical education, have learned how to game these types of tests. By middle school, most students have clear understanding of such test gaming techniques.

The postgraduate medical trainees must pass these board examinations in order to practice their chosen specialty after having spent many years in training and nowadays accumulating considerable debt in that process. They will understandably exercise whatever it takes to be successful on these non-competency-based examinations. Consider that currently virtually all radiology residents will participate in the aggregation and dissemination of “recalls” of questions that are on old examinations a significant number of which will be repeated year over year. The current rubric neither establishes competency nor proves of mastery of a defined curriculum.
 
The system is simply not what it should be for the faculty, the trainee and most of all for the patients. Rather than evaluating competency, the testing rubrics become a “rite of passage” dictated by organized radiology and licensing authorities. The latter authorities are certainly an indispensable part of our profession. As educators, we must help those authorities improve on the product delivered to the public under our professional banner. We have the tools. All we need is the will and effort as educators to make these improvements as soon as possible.
 
It is time to systematically eliminate those errors by developing specific mechanisms of curriculum delivery aimed at eliminating both observational and interpretive errors. Furthermore, our systems must embrace and emphasize those behaviors, which emphasize the appropriate role for diagnostic imaging interpretation interaction with our referring colleagues and patients including, but not limited to, a well-crafted, grammatically correct and fully understandable report. This is our actual work product as diagnostic radiologists and testing these skills should be how we established competency.
 
Dr. Garland told us 70 years ago our error rate was too high 2. Dr. Weed told us 50 years ago that facts are commodity and we needed to educate physicians to think critically and to provide them with smart IT systems that can move them along the most fruitful pathway in medical decision-making 4-8. What progress have we made in response to this excellent advice by two educational pioneers? Clearly not enough progress when reflecting on the lack of improvement in our making potentially significant mistakes.
 
Our next step in diagnostic imaging education should be to disseminate an evaluation rubric that allows us to assure our patients that we are competent. This condition is not satisfied by the current state of, to some extent, arbitrary and haphazard exposure to specific clinical scenarios in an academic clinical practice that supports the training environment during residency and fellowship training. It is also not satisfied by current methodologies of board examinations that really do not effectively evaluate whether the trainees seeking certification can effectively engage in advanced critical thinking at an expert or even competent/proficient level.
 
Modern expressions of competency-based training suggest that a near expert (“proficient”) level of competency should be attained by the end of the training and that true expert level of professional behavior be attained relatively soon after the completion of formal training (Figure 3B-1).  To assure that this occurs, the current methodology of both medical school and postgraduate medical residency training must progress to an eventual fundamentally competency based curriculum and effective and fair evaluation rubrics that do better than currently available tools to assure competency/proficiency/mastery 14. In doing so we must decisively lower what is currently an unacceptably high rate of error in order to truly claim general competency in diagnostic imaging interpretation.


Coming Next:
Part 4:  The Odds of Mastering All Clinical Scenarios in Radiology:  An Invitation to Harm?
 

Anthony A. Mancuso, MD

Dr. Mancuso graduated from the University of Miami School of Medicine in 1973 and completed a Residency and fellowship training in Diagnostic Radiology, including 2 years of subspecialty Neuroradiology training, at UCLA Health. He joined the faculty at UCLA Health where he was fortunate to have a founding member of organized neuroradiology in the United States, Dr. William Hanafee, as his friend and lifelong mentor. Dr. Mancuso owes much of the professional development in his career to Dr. Hanafee both with regard to his dedication to development of effective educational methodology and a devotion to discovery of practices that make a positive impact on patient care. He is a Past President of the American Society of Head and Neck Radiology and Senior Member of the American Society of Neuroradiology.

In 1983, Dr. Mancuso joined the faculty at the UF Health to direct the development of the MRI clinical and clinical research program. In 2000, he became Chairman of the Department of Radiology and remains in that position currently. He is also the President of the Florida Clinical Practice Association at UF Health.

Dr. Mancuso is an acknowledged international expert in the area of ENT radiology having been recognized for his achievements by Gold Medals from the American and European Societies of Head and Neck Radiology and a Presidential Citation from the American Society of Head and Neck chirurgery. He has over 170 refereed publications most in the area of Head and Neck Radiology, and has written several books, most recently, a comprehensive 3 volume text covering the clinical practice of head and neck imaging.

Dr. Mancuso's current research interests have been in developing novel methodologies for radiology education, exploiting foundational and modern learning techniques and merging those techniques with IT tools that make personalized, asynchronous delivery of an effective Radiology curriculum finally possible. His clinical research interest is now focused on the development of advanced brain MRI utilizing DTI and fMRI for the evaluation of traumatic brain injury and a wide range of neuropsychiatric disorders.
 

 References:
 
1- Stephen Waite, Jinel Scott, Brian Gale, Travis Fuchs, Srinivas Kolla and Deborah Reede: Interpretive Error in Radiology AJR 2017; 208:739–749
2- Garland, in groundbreaking work in 1949 Garland LH. On the scientific evaluation of diagnostic procedures. Radiology 1949; 52:309–328 9. Berlin L. Accuracy of diagnostic procedures: has it improved over the past five decades? AJR 2007; 188
3- Berlin L. Accuracy of diagnostic procedures: has it improved over the past five decades? AJR 2007; 188:1173–1178
4-Weed, L. L. (1964-06-01). "MEDICAL RECORDS, PATIENT CARE, AND MEDICAL EDUCATION". Irish Journal of Medical Science. 462: 271–282.
 5-Weed, L. L. (1968-03-14). "Medical records that guide and teach". The New England Journal of Medicine. 278 (11): 593–600. doi:10.1056/NEJM196803142781105. ISSN 0028-4793. PMID 5637758.
6-Weed, L. L. (1968-03-21). "Medical records that guide and teach". The New England Journal of Medicine. 278 (12): 652–657 concl. doi:10.1056/NEJM196803212781204. ISSN 0028-4793. PMID 5637250.
7-Weed LL. Medical records, medical education, and patient care: the Problem-Oriented Medical Record as a basic tool. 1970. Cleveland (OH): Press of Case Western Reserve University.
8-Jacobs L. Interview with Lawrence Weed, MD—the father of the problem-oriented medical record looks ahead [editorial]. Perm J 2009 Summer; 13(3):84–9.
9- Lori Deitte, MD Vice Chair for Education – Department of Radiology, Vanderbilt University – personal communication
10-Knowles, M. (1984). The Adult Learner: A Neglected Species (3rd Ed.). Houston, TX: Gulf Publishing.
11-Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development (Vol. 1). Englewood Cliffs, NJ: Prentice-Hall.
12-Bloom, B., Englehart, M. Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York, Toronto: Longmans, Green.
13 - Sistrom, et. al. Unpublished data
14- AMA.org- Education-Creating the Modern Medical School.