Quantity and Quality of Mind

Table 1

I had to relate the graphic in the last post to the four ways multiple-choice tests can be scored when students pick the questions to report what they know and can do. This is not an efficient way to do standardized testing.

It is a way to promote student development from passive pupil to self-correcting scholar; the basis for high standardized test scores.

Table 1 points out respective pass/fail scores (red box) for the four quantity to quality ratios. The 60% cut point was used in my quantity and quality scoring.

Students voted to value quantity and quality equally. This removed a variable between traditional scoring of marks influenced by their luck on test day; and quantity and quality scoring of marks influenced by their judgment of what they already did know and could do, and of what they had yet to learn: know and understand.

It was also important not to present something that new students would see as a barrier to switching from luck to reason. As is, it took until the third bi-weekly test in each semester for over 90% of students in a class to make the change in scoring and several more weeks to change study habits (bicameral to introspective mind, see previous post).

Chart 1

Chart 1 graphs the above table. It shows the quality vectors for a score of 60%. A traditional test would require a minimum of 30 right marks.

Quantity and quality scoring requires a minimum of 10 right marks added to a perfect 50% value for self-judgment. This rarely happened out of over 3,000 students.

The standard of 90%, set for high risk exams (nuclear power operators, police, and doctors), starts with a 75% value for judgment. You better know what you are doing before you act, or ask questions, or do not answer.

With the active start score (75%) set higher than the cut point (60%) there would be no incentive to mark anything on such a test. A game rule is needed: A minimum of 10 questions must be selected [(75%-60%)/(75%/50 counts) = 10 counts] to include the 60% cut point.

The chart points out that any cut point can be reached from any starting score. Then create a test bank of items with difficulties in the 20s, 40s, 60s and 80s.

Deliver the test over the Internet starting with a question from the 40s or 60s. A test, starting with a few items averaging at 60% difficulty, followed by 10 more items near 60%, may stop and call you a winner at 60%. The test stops when answering more questions is expected to not make a change in the score.

The test is efficient. It takes the least time with the fewest questions. It can dynamically cruise across the four, on paper, static levels to only deliver a few questions that match each student’s preparation instead of presenting a test booklet of a 100 questions.

We will never know what you actually know. But you know enough. We will know that your performance is a near perfect match to the average scores of a select group of individuals that you are believed to be a good fit. This works for graduation, job placement and entrance exams.

It works for high quality students; who may only attend class on test days. Low quality students need a teacher to attend to specific problems.

The Internet statistics are based on all the students tested. Classroom, teacher created test, statistics are based only on that one class. The paper and pencil methods we found so effective are now in the past. The bicameral/introspective mind theory hints at why they worked.

Student Minds

My reading of “Two Origins of Consciousness” by Bill Rowe, Chapter 11 in Gods, Voices and the Bicameral Mind: the Theories of Julian Jaynes, edited by Marcel Kuijsten, 2016, brought back to mind how underprepared college students performed when multiple-choice (forced-choice) was not scored by just counting right marks but by scoring both knowledge (quantity) as well as judgment (quality).

The new science building (1980) contained lecture rooms, spanning two floors for 120 students, that replaced the 64-chair lecture room. I wanted to know what these students actually believed, knew, and understood, in order to do the best job teaching. Bluebook essay exams were no longer feasible.

The time-honored solution in mass testing was, of course, multiple-choice. But this time it was different. Now the class voted to score judgment (quality) as well as knowledge (quantity). Quality was valued the same as quantity.

The unexpected result was passive pupils (with a declared negative interest in the course), in general, became active scholars. They started studying for their own enlightenment rather than memorizing nonsense for the next test.

This change in attitude and how they used their brain in studying; learning by questioning and relating, yielded higher scores in the course and their other courses. Students voted to use knowledge and judgment scoring in other large general studies courses.

Does knowledge and judgment scoring provide a cheap, and easy to use, window into the bicameral and conscious introspective mind? The scoring reinforces successful judgment even if the test score is low (Scores 1-4).

The judgment score, or feel good score, that corresponds to notes that teachers make when scoring an essay test, is key to directing student study behavior. Improve a test score with a high judgment score by studying, making sense, of more material. Improve a test score with a low judgment score by changing study habits.

Or is the case more of practice until the mind changes from bicameral to introspective? We can see and test for the results but we have yet to see what goes on in the brain and in the mind.

One student commented that he had failed the first three exams on which he chose traditional scoring. He elected knowledge and judgment scoring and ended up with a comparable score.

His judgment score (quality) was as bad as his knowledge score (quantity). OH!! “Had I just marked questions I could have used to report what I really knew (100% judgment (quality) score) I would have passed the test. Yes!

Most students made the transformation within one semester. Over 90% switched from traditional scoring to knowledge and judgment scoring after two tests.

Another reported that the teacher is to write the questions, score the test, and tell the student how many right marks were counted. “This is a general studies course that I am taking only to satisfy graduation requirements (pass/fail).”

To me this represents the bicameral mind at work. They are unable to generate an interest for a semester. Their study habits are grade school memorize and forget. Their standing at the end of course is the same as before taking the course (except a general studies course has been checked off).

Knowledge and judgment scoring quickly identifies these students. They elect not to use their judgment. They fear making a wrong mark but feel comfortable gambling (God’s will) for right marks by marking all questions.

They lack the habit of relating information into a web of relationships (making sense) that promotes understanding. They lack the ability to verify what they know.

Test scores can be viewed from both the test maker’s and the test taker’s viewpoints. The test takers can be ranked on knowledge and judgment, quantity and quality. High quality students (scholars) can be expected to do well in the future.

Test makers receive an item analysis that groups questions into four groups based on how students used the questions to report what they believe, know, and understand:

1. Expected:         Most students elect to mark and are right.
2. Misconception: Most students elect to mark and are wrong.
3. Discriminating: Few students elect to mark and are right.
4. Unexpected:    Few students elect to mark and are wrong.

I placed the same misconception items on the next test three times in a row after reviewing the rational for the misconception each time. On each test there were fewer students marking. The class knew they did not know but could not mark the right answer. What we learn by association (growing up with it) is difficult to change by rationalization (introspection).

Discriminating questions provide a powerful insight into what a class understands. In my experience this is better and more reproducible than essay question in large classes.

These student’s brains did not change during the course, but how they used their minds did.

Struggling students could see a validation of their judgment. They did have judgment. They did know what they knew. They did not need to have someone else tell them. They could now see, by way of the judgment score, what scholars do to be successful. Something that is meaningful is easy to remember.

They did not have to fear courses that were cumulative. The answer to that question on the first day was a trip to the registrar’s office for several students each semester.

Introspection was rewarded. Is this a quantifying model for viewing how students use their minds and change their use for the better? It works, whatever the underlying reasons.

The concept of the bicameral/introspective mind has given knowledge and judgment scoring an extended life, possibly in research projects. There is much more to be learned if one choses to work outside of safe areas in psychology and education.

Also see http://residentialcarefortwo.blogspot.com for other related details.

Your comments are appreciated.