Thursday, January 28, 2016

An ASQ Exam Review Workshop Survival Guide

So you’ve been invited to an Exam Review Workshop. Congratulations! You've done your pre-work and made it to Milwaukee. Now, here’s what to expect.

The conference room that will be home for the next two days is just to the left of the wall of photos of all the past ASQ presidents and the display case with Dr. Deming’s Red Bead Experiment. The room itself is huge, with a wall of windows looking out at the river below and a view of the lake in the distance.

After an initial greeting from the Exam Developer, everyone gives a brief introduction in turn. If you are anything like me, something as simple as introducing yourself to a group of strangers might make you nervous.  Add to this the fact that among the 11 other professionals in the room, there might be a PhD statistician, a VP of quality, a twenty-year veteran and a MBB at a major corporation. You could be quite intimidated by the time your turn comes. Don’t be. Remember that as an ASQ certified SSBB, you have as much right to be in the room as anyone else, and you will have a unique set of experiences to draw on.

You were actually chosen on purpose:  participants are invited so that they represent different geographical locations, different job titles and different industries. It might be that you are one of only a few participants who work in a heavy manufacturing setting, or in healthcare, banking or government. Each of you will bring your perspective to the exam, resulting in a more robust product in the end.

The ASQ Exam Developer will give a 30 minute presentation on the entire exam development process (more on this in a later post), and will go over exam review procedures.  You will learn the language of test development in which developers refer to questions as “items,” choices as “options,” the right answer as the “key” and the wrong answers as “distractors.”  After one day of constantly hearing these specialized terms, you will be fluent.

Grab a coffee, take out your exam paper and get comfortable, because the hard work of item evaluation then begins. The developer has collated all the results from the exam that the group took at home earlier in the month. Each item that had 3 or more key disagreements (a nice way of saying 3 or more folks in the workshop got the answer wrong), or 3 or more comments gets flagged. The group as a whole reads through each of these questions and tries to determine if the item needs more work and why. If an item needs work, it is put in a parking lot for a small group to later examine it more thoroughly and attempt to improve its shortcomings.

For example, let’s say that item 16 is flagged for key disagreement.  Item 16 has option A as its key, but 3 participants got the item wrong. One chose B, and two chose D.  This is where we leave our pride behind for the good of the SSBB exam. The three who chose the incorrect answer then explain to the group why they chose what they did. One might have read the question too quickly, and sees now that A is clearly the key. The others might state that they were weak on that particular area of the body of knowledge, but see no problems with the question itself. The group then looks at the performance statistics of the item from the last time it was used in an exam. If there are no red flags, the group as a whole might then decide that item 16 is fine and move on.

Let’s say item 19 is flagged due to key disagreements and comments.  Six people correctly chose the key of B, but the other 6 all chose the same incorrect answer of D. Now the group tries to determine what made distractor D so appealing. Comments attached to the item indicate that 4 people thought that B or D could have been keyable. Another stated that the answer depends on which industry one works in. This item looks like it could use more work to make it less ambiguous, so it is assigned to the parking lot for a small group to tackle later in the day.

Along the way, the group makes sure that there are no overlapping items, that no item cues the answer to another item, and that no one topic is over-represented in the exam (how many principle components questions can one person be expected to answer anyway?).

There are also exam items that have been flagged by the ASQ test developer for housekeeping reasons. For example, an item might not have a complete rationale (an explanation of the answer key, as well as the reasons why the distractors are incorrect, accompanies each item in the database). Each item also has a reference text and page number associated with it. When a new edition of a reference is published, the item reference must be updated as well.

The large group review process continues until all flagged items in the exam have either been approved as-is, or put into the parking lot. In between, ASQ provides a lunch buffet, ample coffee, water and soda, along with the much appreciated cookies to combat the mid-afternoon energy slump.

As you might imagine, the large group session could easily become dominated by a few outspoken individuals. This is where the exam developer, and to a lesser extent, the exam chair, watch to make sure that everyone has a chance to voice their opinions and concerns about items. You might hear the developer say, “We haven’t heard from Sally in while. Sally, what answer did you choose for this item?” Or, “Bob makes an interesting point, but let’s hear from some others who also commented on this item.” The exam developer facilitates to make sure that he gets the best information from all 12 participants, while at the same time addressing the test items, typing changes, putting items in the parking lot and keeping everyone happy and on track. It is a thing of beauty to witness, really.

After the large group has addressed the flagged items, small groups are assigned and given parking lot items to investigate. Actions needed might be to double check the body of knowledge designation for an item, to write the rationale for the distractors for an item, to update the reference for an item, to check the key, to change distractors to make them more (or less) distracting, complete with rationale, or to rewrite the stem of an item to make it less ambiguous.  Sometimes an item needs a complete overhaul, in which case the small group may decide to switch it out for another, similar item in the database.

After the small group work is completed (usually by mid-morning on the second day), the large group reconvenes to review all the changes the small groups have made. Each small group presents their changes, and the large group either approves, tweaks, or decides to replace the item. By 3pm on the second day, the original exam has changed substantially, and the items have been reviewed for clarity, correctness, and consistency.

And what have you gained?  By 3pm of the second day, you will have gathered business cards and sent LinkedIn requests, eaten cheese curds and talked more about rational subgrouping and affinity diagrams that you thought possible. You also will have succeeded in producing a superior certification exam for the next set of SSBB candidates.  It is a very powerful, and empowering, experience. In fact, more than 90% of participants state in their workshop review that they want to be asked back.

Interested? Please send your resume to Mary Martin at ASQ, you must be a member of ASQ and hold a current certification.

© 2016 Mary McShane-Vaughn

No comments: