Analyze and Revise Test

Test questions that follow all the guidelines discussed earlier may still have problems. Problematic questions are not necessarily the ones that students complain about most. A more systemic approach is needed to analyze the test questions.

Item Analysis

An item analysis provides the systematic approach to examine the tests to determine if individual questions function the way they were intended (Worthen et al., 1999). The analysis may be more appropriate for multiple-choice and true/false questions. Worthen et al. suggest the results from the item analysis be used along with the professional judgement to analyze and revise the test, guided by the following questions.

What percentage of students pass or fail a specific question?
Do some question options get selected more or less frequently than expected?
Are there any questions that high scorers do poorly and/or low scorers do well?
Are there any questions that appear to be generally misunderstood, responded randomly, or miskeyed?

Here is how to perform the item analysis using an example from Worthen et al. (1999, pp. 266-277).

Example: An American History test using the multiple-choice format is given to 32 students. The following are steps to conduct item analysis.

Grade the test. If you use a system, the grading may be done automatically.
Rank students' performances from the highest to lowest.
Keep ten highest scores in one group (high-scoring group). Use around 25% of the class as a reference.
Keep ten lowest scores in another group (low-scoring group).
Exclude the rest of the scores from further analysis.
For each test question, count and write down the number of students in the high-scoring group who selected each question option (see example below).
Do the same for the students in the low-scoring group.

Multiple-Choice Question	Item Analysis
Who was the most influential in convincing the French to assist the Americans during the Revolutionary War with England?	Number of high scorers (HS) choosing each option (*10 students*)	Number of low scorers (LS) choosing each option (*10 Students*)
a. George Washington	0	1
b. Marquis de Lafayette	1	4
c. Benjamin Franklin (*correct answer*)	8	2
d. Benedict Arnold	0	0
e. Count de Rochambeau	1	3

No student from the HS group chose George Washington, but one student from the LS group chose that option. Eight students from the HS group chose Benjamin Franklin, but only two students from the LS group chose that option.

In this example, ten students (8 from HS and 2 from LS) chose the correct answer. Many from the HS group got the answer correctly. Many of those in the LS group did not select the correct answer. This is what you should expect from a good question. If most students from the HS group missed this question, the question may not function the way you've planned. If most students in both groups get it correctly, the question may not distinguish HS from LS well. So, the item analysis provides useful information to help you decide whether a revision is needed.

Compute the difficulty level of the question.
Compute the discrimination index of each question or question option.
Evaluate the efficiency of each question option.

If you use the Canvas learning management system, the item analysis information is available, including the difficulty level and discrimination index, for the multiple-choice and true/false questions. Here are the steps to get the information: Go to a quiz >> select Quiz Statistics >> check each question for the difficulty level and discrimination index. (if you select Item Analysis >> download the CSV file >> look for the Difficulty index column and the Point Biserial of Correct column for discriminant index.)

The analysis is not available for other question types such as: matching, essay, fill in blank(s), multiple dropdowns, numerical answers.

Practice

Choose some of your multiple-choice and/or true/false questions and practice computing the difficulty level, discrimination index, and efficiency.

Reference

Worthen, B. R., White, K. R., Fan, X., & Sudweeks, R. R. (1999). Measurement and assessment in schools (2nd ed.). New York, NY: Addison Wesley Longman.

What's Next

Complete the workshop evaluation.