President's Message: Stephen I. Wasserman, MD
ABAI's Secure Exam — Focusing Future Learning Efficiently and Effectively
by Stephen I. Wasserman, MD
I have just returned from a stimulating and important meeting held by the American Board of Medical Specialties (ABMS) focused on Parts II (Self-Assessment) and III (Cognitive Expertise) of the member boards' Maintenance of Certification (MOC) programs. Speakers with quite varied backgrounds (i.e., anesthesiology, internal medicine, infectious diseases, ophthalmology, and medical education) provided a wide perspective on where we are now and where we, as physicians dedicated to the improvement of performance in practice, need to be to optimally address the issues surrounding these two major components of MOC. We learned that physicians, like other professionals, are not particularly good at identifying our own areas of high knowledge and expertise or those areas needing improvement. It is a part of human psychology to overrate our abilities and performance. Additionally, we have learned that when medical errors under physician control are scrutinized, one of the most important deficits found is not knowledge per se, but in the ability to bring together disparate pieces of knowledge and to integrate them for example, to make a difficult diagnosis or prescribe a complicated regimen of care. Given this reality, what are the best strategies to help physicians identify areas of opportunity for improvement?
Despite our personal inability to accurately identify areas of deficiency, it is still useful for us to work on those areas in which we do feel we are not up to speed. But to best supplement our own individual assessments what are good strategies for identifying gaps in knowledge or performance? One strategy is to engage in Part IV MOC activities designed to assess performance in practice. Individual performance in these exercises is then compared to that of our peers, and it rapidly becomes clear in which areas our performance needs improvement. Some issues identified are those of the systems in which we work, but some are clearly in domains under our control including knowledge of latest practices and best strategies to improve medical outcomes. An even more useful tool however is to engage in a comprehensive assessment of our base of knowledge—best exemplified by the secure examination (Part III of MOC requirements). We at ABAI recognize that this is a stressful and not widely beloved aspect of the MOC program but it is by far the best way for physicians to identify areas of weakness. The goal of the secure exam is not to find those who are inadequate but to enable test takers to find out just where they are strong and where there are opportunities for improvement. That is why the questions are purposely selected to be difficult. We at ABAI do know that everyone taking the examination has already passed at least two prior board examinations (their primary board and their initial ABAI certification examination) and has thus previously demonstrated an acceptable knowledge base. Therefore the passing score is set in recognition of this fact and it is possible for everyone to pass—there is no required failure rate. However, it is important to note that there is accumulating evidence that one's performance on such standardized tests deteriorates with time and that scores do drop over the lifetime of a physician. We continue to believe that by far the best use of the secure examination is to identify those areas in which knowledge base may have slipped, or to identify those areas in which one’s performance, although acceptable, is not on par with our professional peers. ABAI is developing a
process by which we hope to convey to takers of the secure examination their own performance in multiple domains, and to compare that performance with all takers of the examination. In that way it should be possible to find areas of strength and relative weakness. Rather than criticize individual physicians for poor performance, it is the goal instead to help focus physician learning over the next few years of the MOC process on those precise areas identified as offering opportunities for improvement. Thus one should view the secure examination not as a summative exercise, reflecting the end of a process, but as a formative one providing insights for focusing future learning. In fact, in the future, it may well prove most efficient for our diplomates for the ABAI to offer the secure examination in the first 1-3 years of the MOC cycle rather than in years 8-10, to more effectively provide a platform for planning productive, self-directed learning for the rest of the MOC cycle. No such plans to alter the examination timing are now under consideration, but it is a topic which I personally would like to hear feedback from you, our diplomates. Again, we at ABAI acknowledge the stress of the secure examination, have set the pass score according to our recognition of prior examination passing performance and thus expect very high pass rates. However, we have made the difficulty of the examination questions such that we all can learn our strengths and weaknesses enabling us to best focus our self-directed learning efforts efficiently and effectively.
Please share your thoughts with me on this topic at abai@abai.org or swasserman@ucsd.edu