Wednesday, August 3, 2016

Copy Detector - RMS

This copy detector is an auto-pilot version of the original cheat checker that could point to the source person. Here (Sheet 8) a pairing index ranks answer sheets by the degree of pairing (Sheet 9).

An interesting feature of this right marked scoring (RMS) nursing test is that Unique pairings (red)  occurred only toward the end of the test. 

There is no distinct break in the beginning of the pairing index and at the end of the pairing count plots to indicate cheating.

Wednesday, July 20, 2016

Test Marks with Student ID by Item

A chart showing marks with student ID by item number is a classic printout for multiple-choice tests for use in class discussions. It is simple, but it lacks the analysis results.

Grade book software can import these files.

This post, #20, ends the pages from the website.

The real magic is using these printouts is in pointing out where students and teachers should spend their time most productively. 

Detailed analyses can be found in Multiple-Choice Reborn and Rasch Model Audit. With the end of NCLB and CCSS, fertile soil may yet be found for knowledge and judgement scoring. It is time to do multiple-choice right. Give students the option of Smart Testing along with traditional Dumb Testing.

Nursing Right Mark Score (RMS)
Biology Knowledge and Judgment Scoring (KJS)

Wednesday, July 13, 2016

Guttman with Scores by Item Difficulty

The lowest scoring student is on the bottom line. The most difficult item is on the right side. The lower right corner is as bad as things can get. The upper left corner is as good things can get.

Again, knowledge and judgment scoring (KJS) has more information to work with (accurate, honest, and fair) than right mark scoring (RMS). The quality score (%RT) is in the 80%s all the way down to a student test score of  about 60%. 

Most of these students actually know what they know and what they have yet to learn. They have a solid basis for learning more. KJS promotes student quality.

Nursing Right Mark Score (RMS)
Biology Knowledge and Judgment Scoring (KJS)

Wednesday, July 6, 2016

Test Maker Counseling Matrix - RMS

The test maker view of a right mark scored (RMS) test is based on item difficulty (%), discrimination ability (A,B,C,D), and item performance (mastery, unfinished, and discriminating).

There is no right mark score (RMS) test taker student counseling matrix as students have no vote on which items to select for their individualized test.

This is a traditional item analysis plus a ranking based on how the item performed on the test. For example, two items have a difficult of 50%. One is unfinished (the entire class is having trouble, or there is a problem with the item or instruction). The other ranks at the highest for discrimination ability (one group knows or can do something that the rest in the class do not know or cannot do).

Students of all abilities missed the first item; mainly lower scoring students missed the second.

Right Mark Scoring (RMS)

Test Taker Counseling Matrix - RMS & KJS

The right mark score (RMS) student test taker counseling matrix for biology is the same as for nursing in the prior post. Again there is another example of two items with the same difficulty (58) but classified differently: unfinished and discriminating. Question 50 was a tally item. 

The knowledge and judgment scoring (KJS) student test taker counseling matrix for biology presents a student view of the test not possible with just RMSing. 

  • (E)xpected             most marked & most right
  • (G)uessing             few marked & few right
  • (M)isconception      most marked & few right
  • (D)iscriminating      few marked & most right

As students select items for their individualized tests they are also voting for item performance. Item 50 is an example of a tally that is not scored for a grade.
Item 6 (58%) was labeled unfinished by the test maker view and here is labeled the only misconception by the test taker view.
Both sets of data  can be sorted (mined) for a variety of relationships. One is to look for copying.

Right Mark Scoring (RMS)

Knowledge and Judgment Scoring (KJS)

Wednesday, June 29, 2016

Test Performance Profile

The Test Performance Profile (see below) was invented to guide item, test, student, teacher and instruction development. Item, Percent Difficulty, and Discriminating are listed in each column sorted first by item and then by difficulty for Master/Easy, Unfinished, and Discriminating.

The high scoring RMS nursing test has far more mastery items than the other two categories. The KJS biology test has far more discriminating items then the total of the other two categories. It also has item 9 flagged as BAD. It may need to be dropped and the test re-scored.

Low scoring Unfinished and highly Discriminating items need to be discussed in class. This information is examined in greater detail in the Student Counseling Matrixes.

Standardized tests do not contain Mastery/Easy and Unfinished items. The goal is to obtain the needed distribution of scores with the fewest items. Discriminating items have a far higher Avg PBR (0.39 and 0.40) than Mastery (which has near zero) and Unlimited (0.12 and 0.15).

Test reliability or reproducibility is estimated by KR20 and alpha. It increases with the length of the test and with discriminating items.  

The Discriminating, 50 item and 100 item values for reliability, are surprisingly close for these two tests with very different students (nursing and general biology), student preparation, and assessment (RMS and KJS). In summary, teacher skill takes precedence over statistics in selecting questions for a test. 

The nursing test measures mastery. The biology test measures the different things that students found of interest in reading assignments and other course actives: lecture and laboratory. The biology students did a good job in picking items to report what they knew and what they had yet to learn (only 9 Unfinished). 

The two columns for Unfinished and Discriminating for biology may look similar to those for nursing if the biology students were forced to guess. The practical, useful, details are in the Student Counseling Matrixes.

Right Mark Scoring (RMS)

Knowledge and Judgment Scoring (KJS)

Wednesday, June 22, 2016

Test Fitness

The average item difficulty on the right mark scored (RMS) test was 84%. That is how well the students prepared for the test. 

The test fitness, the average estimate of the Minimum and Maximum number of answers marked for each question (2.2) is how well the test fit student preparation. Test fitness is then 46%. The test design value of one out of four or 25% is lower than the test fitness estimate.

This test functioned close to a true/false test. The test fitness is the average test score when students discard wrong answers they know are wrong and then guess for a right answer from the remaining items. Knowing and guessing, quality and quantity are intermingled .

Right Mark Scoring (RMS)

The average item difficulty on the knowledge and judgment scored (KJS) test was only 73%. Test fitness was 3.1 marks or 32%. The test design value of 1 out of 5 or 20% is again lower than the test fitness estimate. 

Multiple-choice tests are easier than their design values. The KJS tally analysis indicates the complex make up of the average test score. Only 16% of the items were answered by just guessing. Some 35% were recording mastery. Some 43% of the items each split the class into two groups in which one group did significantly better than the other group. What, who and why? There was only one misconception item on the test at the end of the semester.

With KJS knowing and guessing are clearly identified. Quality and quantity are assessed independently. This can be summarized on a single page: Test Performance Profile.

Knowledge and Judgment Scoring (KJS)