Examination Development Administration & Reporting
Introduction
The American Board of Physician Specialties (ABPS) develops examinations for each of the medical specialties represented by the ABPS Boards of Certification. The examination development process is collaborative, involving ABPS Diplomates, as well as American Board of Medical Specialties (ABMS) and the American Osteopathic Association Bureau of Osteopathic Specialist (AOABOS) Diplomates, who serve as the subject matter experts.
For each specialty, ABPS develops a written certification examination consisting of multiple-choice questions. For several specialties, ABPS also develops oral examinations that provide candidates an opportunity to demonstrate their knowledge and skill in diagnosing and treating patients. The oral examinations permit the candidate to probe for information, develop differential diagnoses, and request laboratory tests or diagnostic examinations.
ABPS also develops multiple-choice recertification examinations. ABPS Diplomates are required to recertify every eight years.
Practice Analysis
All ABPS examinations are based on practice analyses. Meetings with subject matter experts, e.g., ABPS, ABMS and AOABOS Diplomates, lead to the identification of technical knowledge, skills, and abilities essential to the performance of a certified physician. These are then validated through review by a representative sample of professionals in the field, usually certified by the ABMS, the American Osteopathic Association (AOA), or ABPS. Through this process, ABPS determines the importance, criticality, and frequency of the identified knowledge, skills, and abilities required in practice. This information serves as the cornerstone of the content and process validity of the examinations, and ensures that every question on ABPS examinations represents an assessment of knowledge, skills, or abilities clinically relevant to the practice of that medical specialty.
Test Specifications (Examination Blueprints)
The development of blueprints requires the collaboration of professionals with psychometric expertise and those with the medical knowledge. The psychometrician challenges the physicians to consider what skills, knowledge, and abilities can be assessed within the specialty. This approach allows the collaborating team to identify any knowledge, skills, and abilities identified in the practice analysis but not yet addressed by the exam blueprint.
The test specifications include details regarding the weighting of each content category. For example, the test blueprint of each written examination indicates the approximate number of questions for each category and sub-category based upon the data from the practice analysis. The number of topics to be covered, and the number of questions needed to provide reliable diagnostic information in each content category, contribute to decisions regarding the number of questions that will appear on an examination.
The majority of written certification examinations are comprised of 200 items, with the longest examination containing 350. Recertification examinations consist of 100 items. Several of the specialties require candidates for initial certification to successfully complete an oral examination once they have passed the written examination.
Item Development
Using the examination blueprints as a guide, a variety of questions are developed for the examinations. Questions are developed to meet content and task categories. The initial items are designed by subject matter experts, typically ABPS Diplomates, who have particular knowledge or expertise relevant to the examination being developed. Items are reviewed by an editor, who revises the items with attention to item construction rules, readability, grammar, and potential bias. Once the item has been through an editorial review, it is then submitted to a committee of ABPS Diplomates of the same specialty for final editing and approval.
Examination Forms Development
ABPS staff configures each form of each specialties’ examination to meet the needs of the examination blueprint. Subject matter experts then review the draft examination to identify items that are too similar to appear on the same examination form, as well as those that might provide information that leads to the answer of another item. Additionally, an editor reviews the examination form for consistency of style, embedded directions, arrangement of items, and inclusion of all necessary components for a complete examination.
Administration of the Examinations
Examinations are administered twice a year under standardized conditions. Although written examination items and examinations undergo continuing review, the accuracy of an item can be impacted at any time due to changes in accepted medical practice. In order to identify such concerns, candidates have the opportunity to comment on examination items that may be inaccurate or out of date while they are taking the examination. Candidates are advised that their comments will be reviewed before ABPS performs the final scoring and reporting of results.
For oral examinations, specific scripts following a standardized outline are developed for the examiners. ABPS trains the examiners regarding appropriate administration procedures. As with the written examinations, the content and scoring of the oral examinations undergo reviews prior to each administration to verify accuracy and relevance.
Establishing Passing Scores
ABPS uses variations of the Angoff method to establish passing scores on the written and oral examinations.
Review of Comments and Item Analyses
After administration of the written examinations, the candidates’ responses are initially scored and item statistics are generated. The results of the item analysis and a compilation of candidates’ comments are provided to the Examination Committee for review. The candidates’ responses are rescored based upon the recommendations of the reviewers. The results are used to develop the score reports for candidates and examination analysis reports.
Scoring and Reporting
All responses by candidates on the written examinations administered via computer-based testing are submitted and scored electronically. Follow-up audits of scoring procedures and examination results may be conducted. If an audit indicates an error in scoring resulting in a change in the results of a candidate, ABPS will rescore and provide the candidate the revised results.
Recertification Examinations
The written examinations, required for recertification every eight years, are drawn from the same bank of items as the certification examinations. These examinations are administered in two hour sessions and contain 100 multiple-choice questions. The recertification examinations are assessing skills and knowledge applicable to acceptable medical practices in the specialty. These examinations do not address cutting-edge changes in medicine, but rather address established practices that remain current.
Realizing the need for competent physicians to be abreast of the latest developments in medicine, recertifying physicians are required to complete self-assessments in their specialty annually. Physicians may select self-assessments to address their needs and are encouraged to select areas that address current day issues, medical research, and possible changes in medical practice.