About Me

My photo
I am the founder of University Training Partners, a company that designs and delivers online Lean Six Sigma and statistics training. In this blog, I hope to share my thoughts on statistics, quality assurance, and training, drawing from my 15 years of work experience in the field, eight years as a tenured professor teaching graduate statistics classes and ten years as a full-time training developer.

Friday, January 29, 2016

An ASQ Exam Chair Reveals All

What Did You Do Last Weekend?

Last month, I was indoors on a beautiful Saturday afternoon, surrounded by five open statistics textbooks, gripping my calculator as I performed yet another calculation for the ASQ Six Sigma Black Belt exam. I wasn't missing out on my weekend with a group of like-minded quality professionals though. I wasn't even nervously watching the clock: I was sitting comfortably in my home office taking the Black Belt exam in preparation for an ASQ workshop.

We could argue about the sanity of an already certified Black Belt who voluntarily takes the 150-question, grueling exam every six months, but that's fodder for another post. The fact is, twice a year, ASQ randomly chooses 12 certified Black Belts to read through an exam, record answers and, most importantly, provide comments (mostly criticisms) on the questions.  After sending their feedback into ASQ, the 12 are flown to ASQ headquarters in Milwaukee to attend a two-day workshop. There, the group reviews the exam questions and their collective responses and comments, all the while being expertly led through the process by an ASQ test developer. At the end of those two days, the 12 -- by now a cohesive group firmly in the performing stage of team development -- have produced a new version of the exam that is robust, reliable and accurate. The new, vetted exam is then administered to Black Belt candidates on the next test date, either in March or October. It is an extraordinary process performed by ASQ members for ASQ members.


Comments Anyone?

When I sat for my Black Belt exam back in 2006, I had no idea about the amount of work, expertise and caring attention to detail that went into the exam. I was just interested in getting through the test and passing. A couple of the questions had me scratching my head, and one or two seemed just plain wrong. I wrote a few comments on the sheet provided during the test session. Have you commented on exam questions as well? I will tell you that each and every candidate comment is collated into a report and reviewed by the ASQ test developer and the exam chair. (Remember, there are generally 1500 people taking the Black Belt exam throughout the country on test day!)  As you might expect, a lot of the comments are just complaints from cranky test takers. Four hours of filling out a bubble sheet will do that to people. However, it has happened that a test question gets thrown out because candidates have discovered a problem that was missed in the review workshop.


A Way to Give Back

ASQ administers 17 certification exams, from the old stalwarts like the CQE and CQA to the newly created, like the CPGP, or Certified Pharmaceutical GMP Professional. The exams each have at least one review workshop each year, and ASQ is always looking for volunteers. If you are interested in being involved in an exam review, don't wait for the luck of a random draw. Send your resume to Mary Martin at mmartin@asq.org stating which exam (or exams) you are interested in. Volunteers must hold a current certification and be an ASQ member. Participants get 2 REUs, a trip to Milwaukee and the chance to get nerdy with 11 other quality professionals!

Join our mailing list to receive exclusive course discount codes and VIP access to webinars!



© 2016 Mary McShane-Vaughn

Thursday, January 28, 2016

An ASQ Exam Review Workshop Survival Guide


So you’ve been invited to an Exam Review Workshop. Congratulations! You've done your pre-work and made it to Milwaukee. Now, here’s what to expect.

The conference room that will be home for the next two days is just to the left of the wall of photos of all the past ASQ presidents and the display case with Dr. Deming’s Red Bead Experiment. The room itself is huge, with a wall of windows looking out at the river below and a view of the lake in the distance.

After an initial greeting from the Exam Developer, everyone gives a brief introduction in turn. If you are anything like me, something as simple as introducing yourself to a group of strangers might make you nervous.  Add to this the fact that among the 11 other professionals in the room, there might be a PhD statistician, a VP of quality, a twenty-year veteran and a MBB at a major corporation. You could be quite intimidated by the time your turn comes. Don’t be. Remember that as an ASQ certified SSBB, you have as much right to be in the room as anyone else, and you will have a unique set of experiences to draw on.

You were actually chosen on purpose:  participants are invited so that they represent different geographical locations, different job titles and different industries. It might be that you are one of only a few participants who work in a heavy manufacturing setting, or in healthcare, banking or government. Each of you will bring your perspective to the exam, resulting in a more robust product in the end.

The ASQ Exam Developer will give a 30 minute presentation on the entire exam development process (more on this in a later post), and will go over exam review procedures.  You will learn the language of test development in which developers refer to questions as “items,” choices as “options,” the right answer as the “key” and the wrong answers as “distractors.”  After one day of constantly hearing these specialized terms, you will be fluent.

Grab a coffee, take out your exam paper and get comfortable, because the hard work of item evaluation then begins. The developer has collated all the results from the exam that the group took at home earlier in the month. Each item that had 3 or more key disagreements (a nice way of saying 3 or more folks in the workshop got the answer wrong), or 3 or more comments gets flagged. The group as a whole reads through each of these questions and tries to determine if the item needs more work and why. If an item needs work, it is put in a parking lot for a small group to later examine it more thoroughly and attempt to improve its shortcomings.

For example, let’s say that item 16 is flagged for key disagreement.  Item 16 has option A as its key, but 3 participants got the item wrong. One chose B, and two chose D.  This is where we leave our pride behind for the good of the SSBB exam. The three who chose the incorrect answer then explain to the group why they chose what they did. One might have read the question too quickly, and sees now that A is clearly the key. The others might state that they were weak on that particular area of the body of knowledge, but see no problems with the question itself. The group then looks at the performance statistics of the item from the last time it was used in an exam. If there are no red flags, the group as a whole might then decide that item 16 is fine and move on.

Let’s say item 19 is flagged due to key disagreements and comments.  Six people correctly chose the key of B, but the other 6 all chose the same incorrect answer of D. Now the group tries to determine what made distractor D so appealing. Comments attached to the item indicate that 4 people thought that B or D could have been keyable. Another stated that the answer depends on which industry one works in. This item looks like it could use more work to make it less ambiguous, so it is assigned to the parking lot for a small group to tackle later in the day.

Along the way, the group makes sure that there are no overlapping items, that no item cues the answer to another item, and that no one topic is over-represented in the exam (how many principle components questions can one person be expected to answer anyway?).

There are also exam items that have been flagged by the ASQ test developer for housekeeping reasons. For example, an item might not have a complete rationale (an explanation of the answer key, as well as the reasons why the distractors are incorrect, accompanies each item in the database). Each item also has a reference text and page number associated with it. When a new edition of a reference is published, the item reference must be updated as well.

The large group review process continues until all flagged items in the exam have either been approved as-is, or put into the parking lot. In between, ASQ provides a lunch buffet, ample coffee, water and soda, along with the much appreciated cookies to combat the mid-afternoon energy slump.

As you might imagine, the large group session could easily become dominated by a few outspoken individuals. This is where the exam developer, and to a lesser extent, the exam chair, watch to make sure that everyone has a chance to voice their opinions and concerns about items. You might hear the developer say, “We haven’t heard from Sally in while. Sally, what answer did you choose for this item?” Or, “Bob makes an interesting point, but let’s hear from some others who also commented on this item.” The exam developer facilitates to make sure that he gets the best information from all 12 participants, while at the same time addressing the test items, typing changes, putting items in the parking lot and keeping everyone happy and on track. It is a thing of beauty to witness, really.

After the large group has addressed the flagged items, small groups are assigned and given parking lot items to investigate. Actions needed might be to double check the body of knowledge designation for an item, to write the rationale for the distractors for an item, to update the reference for an item, to check the key, to change distractors to make them more (or less) distracting, complete with rationale, or to rewrite the stem of an item to make it less ambiguous.  Sometimes an item needs a complete overhaul, in which case the small group may decide to switch it out for another, similar item in the database.

After the small group work is completed (usually by mid-morning on the second day), the large group reconvenes to review all the changes the small groups have made. Each small group presents their changes, and the large group either approves, tweaks, or decides to replace the item. By 3pm on the second day, the original exam has changed substantially, and the items have been reviewed for clarity, correctness, and consistency.

And what have you gained?  By 3pm of the second day, you will have gathered business cards and sent LinkedIn requests, eaten cheese curds and talked more about rational subgrouping and affinity diagrams that you thought possible. You also will have succeeded in producing a superior certification exam for the next set of SSBB candidates.  It is a very powerful, and empowering, experience. In fact, more than 90% of participants state in their workshop review that they want to be asked back.

Interested? Please send your resume to Mary Martin at ASQ, mmartin@asq.org. you must be a member of ASQ and hold a current certification.

Join our mailing list to receive exclusive course discount codes and VIP access to webinars!


© 2016 Mary McShane-Vaughn

Wednesday, January 27, 2016

Statistical Rules of Thumb and the 3.4 dpmo Dilemma

There are a lot of quality consultants out there touting "quick and dirty" statistical methods: easy ways to get an answer without all that inconvenient math. Short course students leave their Become a Statistician in Two Days classes armed with rules of thumb, quick reference guides, and flow charts of simplified methods. Do these methods work? Yes and no. They do perform as advertised: giving an answer without the rigor. But is the answer the right answer?

My question to you is this: In this age of Six Sigma Quality, where only 3.4 dpmo is acceptable (and remember, that is AFTER the 1.5 sigma shift in mean), how is it that we accept statistical rules of thumb to help us make process decisions?

We claim to wholly embrace this 6 Sigma standard, and then turn around and use critical values of +/-2 for our hypothesis tests, or blindly assume normality and happily proceed with our regression modeling. We measure with a micrometer and then cut with an ax.

It is assumed that statistics -- real statistics-- is too difficult a subject for us quality folks to master. Hence a "statistics without the pain" training industry has sprung up to give us "just the facts, nothing more." Why are we selling ourselves short? Sure, statistics requires math and lots of practice. So what? Aren't our customers and their satisfaction worth a little extra training and number crunching?

Join our mailing list to receive exclusive course discount codes and VIP access to webinars!



© 2016 Mary McShane-Vaughn