What do the KCPE results really mean?

The KCPE is the test taken by all graduating 8th graders in Kenya.  The results determine admittance into high school, and the numbers can also be used to competitively rank primary schools.

In looking again at the numbers, which I had written about before, I realized that I had a lot to learn about these kinds of tests before I could understand what the results really mean.  Before I talk more about the KCPE, let’s look at what is helpful to know first.

As I learned recently, there are two basic kinds of tests: criterion-referenced and norm-referenced.

Criterion-referenced tests are simple to understand.  If I asked you to memorize the fifty US states, for instance, and then had you list them from memory, that would be a criterion-based test.  I’m checking to see if you learned what I asked you to learn.  This type of test is not really designed to pit students against each other… there would be nothing wrong with all students getting 100% on the test.  If anything, it grades the teacher: if all the students did poorly, then maybe I should have done more to reinforce the state names, instead of just asking you to memorize them.

A norm-referenced test is designed to pit students against each other.  They are quite practical.  Take for instance the SAT that most US students take as a college entrance exam.  A college can only accept so many students, so the test is designed to separate out the students as much as possible to make that process easier.   The test is designed so that the scores are as spread out as possible: few get a perfect score, and few completely fail.  If many students got a perfect score, or if many failed, it would be more difficult to differentiate between students at the extremes.

When I wrote about the average Deaf 12th-grader in the US having the reading capabilities of a Hearing 4th grader, that was referring to a special kind of norm-referenced test done by Gallaudet University.  This kind of test is taken first by a “norming” group, and that group sets the standard for everyone who takes it afterward.  In this case, a Hearing group of students of all ages took the test, and the results become the baseline.  If we averaged all the Hearing fourth graders’ results, and the result was 60 points (I am making this number up), then getting 60 points on the test would be considered “fourth grade level” based on the Hearing norms.  So when the average Deaf 12th grader takes the test and gets 60 points, (s)he is considered to be reading at “fourth grade level.”

This test does not help us understand if fourth graders know what they’re supposed to know considering their age.  It’s entirely possible that a criterion-referenced test might reveal that, on average, Hearing fourth graders know exactly what they should know, but Hearing 12th graders are, on average, only familiar with material that they learned in the 8th grade (I’m making this up as an example).  The point is that you can’t compare the two kinds of tests—the statistics get all wacky and apples-and-oranges.  Literacy statistics in particular are all over the place: some look at total population including immigrant population, and some (like the Gallaudet test) look only at currently active students.

So the KCPE.

The KCPE is norm-referenced.  Although no effort is made to distinguish between “grade levels” like the previous examples, is is designed to determine high school placement, which is highly competitive because there are only three Deaf high schools in the country.  So if the test is doing its job, few students will fail, and few will excel.

The lowest possible KCPE score is zero and the highest is 500.  The highest attained score each year is usually in the mid-400s.  It is mostly multiple choice with four options, although there is single composition included in the test that is factored into the English score.  Ignoring the essay, a student would receive a score of 25%, or 125 points, just by guessing “C,” for example, on every single question.  I am not sure how heavily the essay is weighed, but assuming it was left blank, that would bring the score down more.  For the purpose of this conversation, anything in the low-100’s would be considered complete failure, basically the equivalent to filling out the test at random.

The average 2007 KCPE score for Deaf Schools is 123.05.  I’ve heard that it went down for 2008.  Only one Deaf School in the entire county has scored over 140 for two years in a row.  This means two things:

1. The test is not working for Deaf Schools as intended.  When considering a student with an average score, high schools can’t tell the difference between the following two types of students:
   • The student who knows absolutely nothing and guesses at random.
   • The students who actually knows the correct answer to 25% of the test and leaves the rest blank.
2. With one exception, Deaf Schools are doing abysmally.  To not even register on the national test, to be statistically indistinguishable from complete guessing, means that something is terribly wrong.

What is even worse, I think, is that the Deaf Schools are ranked using these numbers, even though they could easily be plotted on a bell curve centered around 123.05, which simply means that some schools had better or worse guessers that year.  COMPLETLEY INGIGNIFICANT DIFFERENCES, and dangerous, because a bad school with good guessers could appear to outperform a better school.

My recommendation for the test (in case anyone from the Ministry of Education is reading) would be some combination of:

• Designing a different test for Deaf Schools to make ranking and admissions more accurate.
• Modifying the scoring system to penalize guessing.

My recommendation for making students do better on the test:

• Well, that’s what keeps me up late, isn’t it?

OK, enough numbers.  Here’s a really grainy photo taken with my cameraphone on Valentine’s Day.  We’re waiting for our delicious Indian dinner.  The girls were given real roses by the restaurant.  No flower for me, though, but the good company made up for it. 🙂