What do the KCPE results really mean?

The KCPE is the test taken by all graduating 8th graders in Kenya.  The results determine admittance into high school, and the numbers can also be used to competitively rank primary schools.

In looking again at the numbers, which I had written about before, I realized that I had a lot to learn about these kinds of tests before I could understand what the results really mean.  Before I talk more about the KCPE, let’s look at what is helpful to know first.

As I learned recently, there are two basic kinds of tests: criterion-referenced and norm-referenced.

Criterion-referenced tests are simple to understand.  If I asked you to memorize the fifty US states, for instance, and then had you list them from memory, that would be a criterion-based test.  I’m checking to see if you learned what I asked you to learn.  This type of test is not really designed to pit students against each other… there would be nothing wrong with all students getting 100% on the test.  If anything, it grades the teacher: if all the students did poorly, then maybe I should have done more to reinforce the state names, instead of just asking you to memorize them.

A norm-referenced test is designed to pit students against each other.  They are quite practical.  Take for instance the SAT that most US students take as a college entrance exam.  A college can only accept so many students, so the test is designed to separate out the students as much as possible to make that process easier.   The test is designed so that the scores are as spread out as possible: few get a perfect score, and few completely fail.  If many students got a perfect score, or if many failed, it would be more difficult to differentiate between students at the extremes.

When I wrote about the average Deaf 12th-grader in the US having the reading capabilities of a Hearing 4th grader, that was referring to a special kind of norm-referenced test done by Gallaudet University.  This kind of test is taken first by a “norming” group, and that group sets the standard for everyone who takes it afterward.  In this case, a Hearing group of students of all ages took the test, and the results become the baseline.  If we averaged all the Hearing fourth graders’ results, and the result was 60 points (I am making this number up), then getting 60 points on the test would be considered “fourth grade level” based on the Hearing norms.  So when the average Deaf 12th grader takes the test and gets 60 points, (s)he is considered to be reading at “fourth grade level.”

This test does not help us understand if fourth graders know what they’re supposed to know considering their age.  It’s entirely possible that a criterion-referenced test might reveal that, on average, Hearing fourth graders know exactly what they should know, but Hearing 12th graders are, on average, only familiar with material that they learned in the 8th grade (I’m making this up as an example).  The point is that you can’t compare the two kinds of tests—the statistics get all wacky and apples-and-oranges.  Literacy statistics in particular are all over the place: some look at total population including immigrant population, and some (like the Gallaudet test) look only at currently active students.

So the KCPE.

The KCPE is norm-referenced.  Although no effort is made to distinguish between “grade levels” like the previous examples, is is designed to determine high school placement, which is highly competitive because there are only three Deaf high schools in the country.  So if the test is doing its job, few students will fail, and few will excel.

The lowest possible KCPE score is zero and the highest is 500.  The highest attained score each year is usually in the mid-400s.  It is mostly multiple choice with four options, although there is single composition included in the test that is factored into the English score.  Ignoring the essay, a student would receive a score of 25%, or 125 points, just by guessing “C,” for example, on every single question.  I am not sure how heavily the essay is weighed, but assuming it was left blank, that would bring the score down more.  For the purpose of this conversation, anything in the low-100’s would be considered complete failure, basically the equivalent to filling out the test at random.

The average 2007 KCPE score for Deaf Schools is 123.05.  I’ve heard that it went down for 2008.  Only one Deaf School in the entire county has scored over 140 for two years in a row.  This means two things:

  1. The test is not working for Deaf Schools as intended.  When considering a student with an average score, high schools can’t tell the difference between the following two types of students:
    • The student who knows absolutely nothing and guesses at random.
    • The students who actually knows the correct answer to 25% of the test and leaves the rest blank.
  2. With one exception, Deaf Schools are doing abysmally.  To not even register on the national test, to be statistically indistinguishable from complete guessing, means that something is terribly wrong.

What is even worse, I think, is that the Deaf Schools are ranked using these numbers, even though they could easily be plotted on a bell curve centered around 123.05, which simply means that some schools had better or worse guessers that year.  COMPLETLEY INGIGNIFICANT DIFFERENCES, and dangerous, because a bad school with good guessers could appear to outperform a better school.

My recommendation for the test (in case anyone from the Ministry of Education is reading) would be some combination of:

  • Designing a different test for Deaf Schools to make ranking and admissions more accurate.
  • Modifying the scoring system to penalize guessing.

My recommendation for making students do better on the test:

  • Well, that’s what keeps me up late, isn’t it?

OK, enough numbers.  Here’s a really grainy photo taken with my cameraphone on Valentine’s Day.  We’re waiting for our delicious Indian dinner.  The girls were given real roses by the restaurant.  No flower for me, though, but the good company made up for it. 🙂


7 Responses to “What do the KCPE results really mean?”

  • damnit – i meant to post the most recent comment over at the most recent entry re: the KCPE – Paul – can you move it?

    • I bought a photocopy of the 2008 KCPE in town. They call them “Mock KCPEs” or “KCPE boosters” or “past papers.” 350 shillings and it’s bound and has answers and explanations. It even includes samples of what the answer sheets (the ones with bubbles) look like.

      I’ve also thought about how test-taking is a skill unto itself, and how knowledge is too often demonstrated here by copying off the board… I haven’t really seen a strong testing culture so far.

  • This is very interesting Paul, I found out this week that my school was ranked #1 in the KCPE, I think one of the student even got a 200 on the test. Everyone seems very proud at my school, even though the school was inspected for its dilapidated state ( walls missing, animals in classrooms, multiple classes in one room.) by the DEO. I had not met the previous class 8, but I do teach the current class 8, and I know that they can not write a complete sentence and seem to answer every question given with any word they know how to spell. I know my school reveling in their success but I know for sure we will not achieve this again. I wonder which school consistently scores high on the test, and if its matter of test taking skills or actual comprehension?

    • Yeah, I saw your school at the top of the list. 🙂 I think you only had five kids or so take the test, so probably one doing well would be able to bring up the average. The ranking details aren’t public so I’m being a bit ambiguous about school names on purpose. Some trivia, though, and a hint: The one school that does consistently “well” actually had an ex-Peace Corps volunteer as its headmaster for many years! I want to go check it out and see how they do things differently.

  • janet robbins

    Oh, my! You certainly have a big job ahead of you. Actually, you could have stayed in the US and taught at McArthur Elementary and you would have seen many of the same challenges in teaching. Here are some ideas that came to mind as I was reading your blogs. Concerning testing, having a sample test is important. Using the testing vocabulary definitely helps the students succeed. Example: On a test students may be asked to locate an example of alliteration in the selection below. Even if the students know what alliteration is, they may not be familiar with the terms such as example, locate and selection.

    In one blog you wrote that students couldn’t write a complete sentence. With little kids, sentences they have dictated are written on a sentence strips (strips of tag board or paper) and then cut into word sections. Use different colored markers or paper for each sentence. The students must rearrange the words to create their sentences. I don’t know if you could apply this to teaching deaf students. I also don’t know if you have paper and markers!
    If you do have large strips of paper, you could write each student’s name on the strips and hold it up when calling on them for a response. This is a good way to make sure you’ve called on every student each day. Make sure you draw from the already-called-upon pile every now and then to keep their attention. It also cuts down on the chalk dust created by writing their name each time you call on them. We wouldn’t want you to come down with a case of white lung!
    That’s it from my bag of teaching tricks for today. I have never taught a deaf student. Good luck.

    • Thanks for the suggestions Aunt Janet. I like the sentence strip idea. I had previously been thinking of a variation of this, in which the the pieces are interlocking so that they can only be placed in a correct location. I thought this would be good for the really young kids, who could just play with it like a puzzle, where the nouns could have cute corresponding pictures. Perhaps the pieces could even be made in the school’s woodshop. Anyhow, the paper strips sound like an easier way to start that would still be fun.

  • i am equally confused about how the marking is done last year 2008 i had a pupil who scored 1oo% maths but the guys from knec decided it was 92 and many more such examples what is going on can somebody please explain?

Comments are currently closed.