A new insight into examinee behaviour in a multiple-choice test: a quantitative approach

Abstract

In the context of both the facility / difficulty index and the examinees’ ability in a multiple-choice item, this study is an attempt to gain a deeper view of these issues in a test targeted to a personnel selection process. Our analysis compares the data supplied by the Classical Test Theory (CTT) and the Item Response Theory (IRT), and follows examinee behaviour on the distractors in the hope that the information obtained through these theories will give us a better insight not only into the item and examinee ability but also into the role the distractors may play in this sort of test. On the basis of the information provided we claim that if the test was intended to measure a desired level for a personnel selection process, the test designer has not succeeded in providing even a single item where the correct option and appropriate distractors fulfill the expected criteria of a good item, at least for upper intermediate level candidates. Since the degree of difficulty is a matter not only of the item but also of the distractors, the test should be thoroughly revised and many distractors modified.