Summary Report on the Development
 of Certification Examinations for
Practicing Firearm and Toolmark Examiners

ABSTRACT

The goal of this project was to develop certification examinations for practicing Firearm and Toolmark Examiners. Certification examinations have been developed for three subject areas within the field of Forensic Firearm and Toolmark Identification. The three subject areas covered by these examinations are: 1) Firearm Evidence Examination and Identification, 2) Toolmark Evidence Examination and Identification, and 3) Gunshot Residue Evidence Examination and Identification. The certification examination for each subject area consists of a written examination and a practical examination. An applicant must pass both examinations to be certified in that subject area. The written examination must be passed before the applicant is allowed to take the practical examination in each subject area. Written examinations are offered at a cost of fifty dollars ($50.00) per subject area and a practical examination will be offered to eligible applicants at a cost of two hundred dollars ($200.00) per subject area. Two separate but equivalent versions of each certification examination have been prepared in each subject area in order to give some longevity to the program. These certification examinations are available to members of the Association of Firearm and Toolmark examiners (AFTE) at the Annual Training Seminars and their local workplaces.

The goal of this project was to develop certification examinations for practicing Firearm and Toolmark Examiners. The Association of Firearm and Toolmark Examiners (AFTE) expects that this will act toward the public benefit by attesting that applicants who pass these examinations have met standards defined by their peers. AFTE also hopes to promote professionalism among its members by providing a recognizable level of accomplishment toward which they can work. This project was supported under award number 97-IJ-CX-0038 from the National Institute of Justice, Office of Justice Programs, U. S. Department of Justice. Points of view in this document are those of the author and do not necessarily represent the official position of the U.S. Department of Justice.

Certification examinations have been developed for three subject areas within the field of Forensic Firearm and Toolmark Identification. The three subject areas covered by these examinations are: 1) Firearm Evidence Examination and Identification, 2) Toolmark Evidence Examination and Identification, and 3) Gunshot Residue Evidence Examination and Identification. The certification examination for each subject area consists of a written examination and a practical examination. An applicant must pass both examinations to be certified in that subject area.

As of June 2000, approximately 45 Firearm and Toolmark Examiners have taken written examinations in these subject areas. An applicant must pass the written examination before he is allowed to take the practical examination in each subject area. Not all the examiners have taken the written examinations in all three subject areas. Forty-three (43) examiners have taken the Firearm Examination; twenty-five (25) examiners have taken the Toolmark Examination and thirty-two (32) examiners have taken the Gunshot Residue Examination. Written examinations are offered at a cost of fifty dollars ($50.00) for each subject area. Presently, the AFTE Annual Training Seminars have provided the vehicle through which the written certification examinations have been administered.

To date, no practical examinations have been given. Some of the final practical examinations have only recently been completed. The practical examinations are expected to be offered to eligible applicants sometime during the first quarter of 2001. They will be offered at a cost of two hundred dollars ($200.00) per subject area. U.S. applicants who have laboratory facilities available to them can have their practical examinations administered at their local workplace. The Bureau of Alcohol, Tobacco and Firearms (ATF) has also offered the use of their laboratory facilities in Rockville, MD as a potential site for the administration of practical examinations. Applicants who do not have facilities readily available to them, i.e. some local, international and independent members, may be able to take their practical examinations at the ATF facilities in Rockville, MD. ATF is presently considering the scope of their offer.

Costs for both the written and practical examinations have been determined such that the certification program can be self-sustaining and affordable. Two separate but equivalent versions of each certification examination have been prepared in each subject area in order to give some longevity to the program.

  SCOPE AND METHODOLOGY

In order to develop content-valid certification examinations that are capable of withstanding potential court challenges, it is essential to use test development procedures that are in compliance with applicable professional and legal standards such as: the Civil Rights Act of 1991, the Americans With Disabilities Act, Age Discrimination Legislation, the Uniform Guidelines for Employee Selection Procedures, the Principles for the Validation and Use of Personnel selection Procedures and the Standards for Educational and Psychological Testing. To ensure that these certification examinations were developed according to the previously mentioned standards, AFTE entered into a contract with Cooperative Personnel Services (CPS) in January of 1998.

CPS is a self-supporting public agency, headquartered in Sacramento, CA, that offers a full range of human resource and related management consulting services to government and non-profit agencies in the United States and Canada. It specializes in occupational testing. CPS currently has more than 1,000 clients throughout the United States and Canada, distributes more than 100,000 tests per year and administers another 150,000 tests per year.

AFTE is a non-profit, professional organization composed primarily of practicing Firearm and Toolmark Examiners. It has over 600 members located in more than 30 countries throughout the world. It was founded in 1969 in Chicago, IL. AFTE publishes its own peer-reviewed journal on a quarterly basis. It also hosts an annual training seminar in a location that varies from year to year. AFTE is the only organization of which I am aware that is dedicated solely to the issues involved in the forensic examination of firearms, toolmarks, gunshot residues, and other related evidence.

Certification has been a subject of contention within AFTE since the late 1970s when several AFTE members served as part of the Criminalistics Certification Study Committee supported by the Forensic Science Foundation under a grant from the Law Enforcement Assistance Administration. Their recommendations essentially died from lack of action and the Criminalistics Certification Study Committee foundered for lack of funding. The subject remained somewhat dormant until the early 1990s when the AFTE Board of Directors established an ad hoc committee to study the feasibility of certification. A somewhat negative summary report was sent to the AFTE Board of Directors by this committee on September 10, 1991, and the issue was shelved again until mid 1993. At that time, an ad hoc committee was formed to monitor the certification efforts, if any, of other professional organizations such as the California Association of Criminalists (CAC), the Northeast Association of Forensic Scientists (NEAFS), the Mid Atlantic Association of Forensic Scientists (MAAFS), the Southeast Association of Scientists (SEAFS), the Southwest Association of Forensic Scientists (SWAFS), the American Board of Criminalistics (ABC), etc. In keeping with the regional nature of the organizations being monitored, the committee was organized such that each committee member was responsible for a specific region of the country i.e. Northeast, Southeast, Midwest, South Central, Northwest, Southwest, and California. It was quickly determined that a certification program for Firearm and Toolmark Examiners had already been initiated by the CAC and that the ABC was considering initiating one also. When this information was given to the AFTE Board of Directors, they changed the goal of the committee from monitoring certification efforts of other organizations to the development of a Certification Program for AFTE. This change in objectives was approved by the AFTE membership at the annual business meeting in 1994.

An extensive Certification Survey was prepared by the committee and published in the AFTE Journal in the fall of 1994 (October). The response to this survey allowed the committee to prepare a Certification Program which was presented to the AFTE membership at the annual business meeting in 1995. The program was voted down by a wide margin but a special mail vote of the membership mandated that the committee revise the plan for future consideration. A revised Certification Program was presented to the AFTE membership at the 1996 annual business meeting. It passed by a narrow margin; however, a motion to fund the program was defeated.

In December of 1996, the committee prepared a grant proposal to fund our program in response to a solicitation from the National Institute of Justice for the fiscal years 1995 and 1996. The grant proposal was approved and the grant was awarded in August of 1997. Requests for proposal were sent to 38 firms engaged in the business of developing occupational and credentialing examinations. Responses were reviewed and, as previously mentioned, a contract was let to CPS in January of 1998 to help AFTE develop and validate a series of certification examinations.

The key to the development of content-valid certification examinations is a thorough and comprehensive occupational analysis. Our job analysis questionnaire consisted of a 33 page booklet that contained 61 job task statements, 153 job knowledge statements, 51 job skill statements and 106 job ability statements. In February of 1998, the questionnaires were mailed to each AFTE member along with a response sheet and a stamped, return envelope. AFTE members were asked to rate the job task statements with regard to: 1) frequency of performance and 2) importance. They were asked to rate the job knowledge, skill and ability statements with regard to: 1) importance and 2) competency level expected at time of certification. The responses were mailed directly to CPS who determined that our response rate of over 33 per cent was more than sufficient to proceed to the next step. Statistics were compiled and all job task statements and job knowledge, skill and ability statements that did not achieve a sufficient rating in each category were eliminated from future consideration. All remaining job task statements and job knowledge, skill and ability statements would have to be covered to some degree by our examinations in order for them to have any validity.

In April of 1998, the regional members of AFTE’s Certification Committee traveled to CPS headquarters to link the surviving job task statements to the surviving job knowledge, skill and ability statements. We also linked the job knowledge, skill and ability statements to the test instruments that we needed to create. These linkages are necessary to demonstrate the relationships between the content of the actual job and the contents of the tests being created. Otherwise, the tests could not be considered to be content-valid. As a result of this meeting, CPS furnished AFTE with a test development plan. It consisted of a list of job knowledge, skill and ability statements that must be covered in each component of the certification examinations.

One other significant event occurred during the April 1998 meeting. CPS personnel instructed the AFTE committee members in the proper development of multiple choice test questions. The workshop was geared toward assisting the committee members in training other AFTE members to write proper multiple-choice questions for the written components of the certification examinations. Each AFTE committee member was asked to return to his respective geographical region and recruit between 10 and 15 AFTE subject matter experts (SMEs) from surrounding states. These SMEs would be responsible for writing the test questions. In this manner, each region of the country would submit potential test questions directly to CPS for each written examination. No one would be responsible for a significant or sizable portion of any of the written tests and, therefore, no one would have an advantage or become ineligible from taking the examinations. All SMEs were required to sign security agreements that provided for severe consequences if they revealed or retained any potential test questions submitted to CPS.

To facilitate the test development process, the lists of job knowledge, skill and ability statements (KSAs) that were furnished to AFTE by CPS were further refined by ranking them according to the ratings they received in the occupational analysis questionnaire. The ranked lists were then given to the regional AFTE committee members who divided them into groups according to how many SMEs they were able to recruit. The ranked groups of KSAs were distributed to the SMEs in each area of the country with instructions to emphasize the higher ranked KSAs when researching job-related materials for potential test questions. Each SME was asked to write 10 to 15 questions in each subject area. They were given until June 30, 1998 to send their questions and reference citations to CPS. CPS was to review them for duplication, content, clarity and form. The edited questions would then be entered into a computerized, test question bank in order to format them into the appropriate examination. Each written certification examination contains 100 multiple-choice questions.

A total of 87 SMEs had been recruited from around the country. If each of them had lived up to their commitment and had written the minimum number of test questions in each subject area (10), it would have generated 870 raw test questions per subject area. Since only 200 usable test questions were needed in each subject area to make two versions of a 100-question examination, we should have had plenty of material. Unfortunately, the SMEs did not live up to their commitments and we had a serious shortage of questions.

Nevertheless, AFTE’s regional certification committee members traveled to CPS headquarters for a planned meeting to design the practical certification examinations. The subject matter, format and scoring schemes for the practical examinations were decided in conjunction with CPS personnel. Volunteers from AFTE’s Certification Committee agreed to prepare prototypes of the practical examinations for further evaluation. We also encouraged the AFTE committee members to solicit their SMEs to honor their commitments and write their full compliment of test questions.

By August 1998, the number of written test questions generated had not improved and the AFTE Certification Committee volunteers had also fallen behind in the preparation of the prototype, practical certification examinations. Therefore, we petitioned the National Institute of Justice (NIJ) for a grant extension from a project end date of December 31, 1998 to a project end date of December 31, 1999. This extension was graciously granted and we began to look at ways to breathe new life into the program.

In November 1998, we obtained a list of KSAs from CPS that had not been adequately covered for the written certification exams. We asked the regional AFTE committee members to select KSAs from the list that they believed their SMEs would be best suited to research. We told them to give one KSA to each SME with instructions to write three or four questions on that KSA alone. We hoped that this technique would reduce the duplication of test questions we were experiencing, and make more

productive use of our SME’s time by focusing their attention in one area. It would also be less intimidating for them to write three or four questions instead of 30 to 45 as we originally asked them to do. This seemed to help the situation and by the end of the year, we had accumulated nearly 1,000 usable test questions in our computerized bank. It was not nearly as many as we had hoped but it was enough to formulate two versions of a written certification examination in each subject area.

Concurrently, by the end of February 1999, the AFTE Certification Committee volunteers had developed two versions of a prototype, practical certification examination in each subject area. The AFTE Board of Directors selected a new panel of subject matter experts (12) from various parts of the country to conduct an Angoff review of the written test questions and pilot test the prototype, practical certification examinations. These panel members were also required to sign security agreements which provide for severe consequences if any test materials are compromised in any way.

In March 1999, this panel of subject matter experts was convened at CPS headquarters. They critiqued the prototype, practical certification examinations which they had pilot tested one week prior to their departure. They also conducted an Angoff review of all the written certification examination questions under the direction of the CPS staff. The Angoff review, which rates the relevancy and difficulty of all the test questions, is essential to establish the equivalency of the different versions of the examinations, and to set the pass point for the examinations. The panel’s comments regarding the prototype, practical certification examinations were generally favorable with the exception of those examinations in the Toolmark subject area and the second component of the Gunshot Residue examination. The Toolmark examinations were judged to be too easy; however, their concept was approved pending the manufacture of new, more difficult unknown specimens. Some portions of the second component of the Gunshot Residue examination had to be redone also. The pass point for the written certification examinations was set at 70 based on the criteria that a Firearm and Toolmark Examiner, who had been through an approved training program with five years experience, should be able to pass the examinations with adequate preparation.

The period from April 1999 to June 1999 was spent preparing and publishing the documents necessary to administer the written certification examinations. These included application forms, suggested study guides, proctor’s instructions, proctor report forms, applicant security agreements, applicant comment forms, and test roster checklists. In addition, appeal process procedures and document retention plans were formulated.

  DETAILED FINDINGS AND ANALYSIS

The AFTE Board of Directors agreed to offer the written certification examinations free of charge to the first 100 eligible applicants at the 1999 Annual Training Seminar in Williamsburg, VA. This was done to induce members to take the examinations at the first offering in order that adequate statistics could be compiled to measure the performance of the examinations. In July 1999, thirty-one (31) AFTE members took the Firearm examination, seventeen (17) took the Toolmark examination and twenty-two (22) took the Gunshot Residue examination. Although the sample size for each of the written examinations is too small to draw any definitive conclusions, it would appear that the examinations are reliable and that the alternate versions are comparable (see Figure 1.) In general, reliability coefficients (RELIABILITY) larger than 0.70 are considered to be an indication of a reliable examination i.e. for sample sizes (N) greater than 100. Tests with smaller sample sizes tend to have smaller reliability coefficients. Therefore, considering our small sample sizes, these reliability coefficients are extremely high. Comparison of the mean scores (MEAN) and the standard error of measurement (SEM) for the different versions of the examinations indicate that they are equally difficult and can be used interchangeably. However, although these initial results are very encouraging, larger sample sizes must be obtained for all versions of the examinations i.e. N>30.

The period from August 1999 to December 1999 was used to review the written certification examinations and the applicant comment forms from July 1999. This time frame was also utilized to manufacture the final editions of the practical certification examinations in the Firearm and first component of the Gunshot Residue subject areas. The review of the written test materials revealed that further editing of the written test questions was necessary. Also, the final, practical certification examinations in the Toolmark and second component of the Gunshot Residue subject areas had not been prepared. These two facts led us to petition the NIJ for one more grant extension. This petition was also successful, and our project end date was extended to June 30, 2000.

The first draft of CPS’s final Report of Occupational Analysis and Certification Examination Development was received in November 1999. It was distributed to selected AFTE Certification Committee members for review and editing. Extensive editing of this report led to several successive drafts. The final version was accepted for distribution to the AFTE Board of Directors in March 2000.

The extensive review of all written test questions that was begun in the last quarter of 1999 was completed in the first quarter of 2000. A list of revised and edited test questions was compiled and sent to CPS for incorporation into the computerized, test question bank. The revised questions will be used in all future examinations.

An ad hoc committee, appointed by the AFTE Board of Directors to determine fee structure and other issues related to the administration of the certification examinations, submitted its first proposal in draft form in February 2000. In April 2000, we received their second draft, and the AFTE Board of Directors established the current fee structure during meetings at the 2000 Annual Training Seminar in St. Louis, MO. It was also decided during these meetings that proctors for the certification examinations should consist of either: 1) AFTE members who are already certified, 2) AFTE members who are ineligible to take the certification examinations e.g. members who served on the certification committee or on the panel that conducted the Angoff review or 3) as a last resort, a non AFTE member of a professional, forensic organization appointed by the applicant’s laboratory director. This last category of proctors was included in an effort to keep costs down and make the examinations more affordable. This was also the impetus for accepting the Bureau of Alcohol, Tobacco and Firearms offer to use their facilities in Rockville, MD as a potential testing site. The specifics of this offer is presently being negotiated between ATF laboratory administrators and the AFTE Board of Directors

The revised written certification examinations were offered to eligible AFTE members at the 2000 Annual Training Seminar at a cost of fifty dollars ($50.00) per subject area. Twelve (12) members took the Firearm examination, eight (8) members took the Toolmark examination and ten (10) members took the Gunshot Residue examination. These numbers combined with those generated during the first offering (1999) yield the totals mentioned previously. Unfortunately, the number of applicants who took the examinations in June 2000 was even smaller than the number of applicants who took the examinations in July 1999. This makes the validity of any statistical inferences for this offering even more dubious. However, the results are included here for the sake of completeness (See Figure 2). Once again, given the extremely small sample sizes (N), comparison of the means (MEAN) and the standard errors of measurement (SEM) for the different versions of the examinations indicate that they are equivalent and interchangeable. Also, the reliability (RELIABILITY) looks very good i.e. over 0.70. Since the written examinations were extensively revised prior to the second offering, the statistics from each offering cannot be combined to yield larger sample populations. The next time the written certification examinations will be offered is at the 2001 Annual Training Seminar in Newport Beach, CA.

  DISCUSSION

The establishment of the fee structure and the administrative procedures for these examinations clears the way for the offering of the practical certification examinations. This is now scheduled for the first quarter of 2001. AFTE Certification Committee Chairman J. E. Murdock, who is responsible for the production of the Toolmark Practical Examination and the second component of the Gunshot Residue Practical Examination, has stated that all components of the practical examinations will be completed by November 2000. Documentation pertaining to the administration of the practical certification examinations will be completed by January 2001.

The written certification examinations have already been successfully offered two times to AFTE members via the Annual Training Seminars. They are scheduled to be offered a third time at the 2001 Annual Training Seminar. The AFTE Board of Directors needs to decide if there will be alternate methods of administering these examinations and what those methods will be.

This successfully concludes our project although I would strongly urge the AFTE Board of Directors to consider establishing a mechanism to evaluate whether or not these examinations are fulfilling their design criteria i.e. can the examinations differentiate between: 1) examiners possessing five years experience, an approved training regimen, and adequate preparation, and 2) examiners who do not have these credentials.

Kenneth Kowalski, M. S., is a semi-retired Criminalist at the Arizona Department of Public Safety, Phoenix, Arizona.

The author would like to thank J. E. Murdock and W. H. Morris Jr. for their comments on an earlier draft of this report.