In August last year I wrote how Professional accountancy bodies believe that competency can be measured by a candidate scoring 50% and failure if scoring 49%. This all or nothing approach seems neither fair nor useful; hence the idea that grades of competence could be introduced e.g. 50% pass, 65% commendation, 75% distinction etc.

But the mark a student gets in an exam is only part of the story when it comes to measuring competence. Can a student be considered competent if the exam they pass only includes questions from say 75% of the syllabus? Yes the whole syllabus might be covered in an 18 month period but in any one exam 25% is not tested. Equally the 75% is often considered core and so examined every sitting, this means that a student need only focus on the 75%. Admittedly if the pass mark is 50% they need to score 50% out of 75% (67%) but with practice this is possible. One final observation, it is unlikely the student will score 0% on the non-core part of the syllabus. They may get say 5% or even 10% out of 25%. Not a great score but the 67% pass mark now becomes 58%. This logic sits at the heart of the exam driven approach.

Objective testing might be the answer?

Objective tests (OT) – test that are relatively short and can be unambiguously marked, are considered by some to be a weaker form of assessment, they are part of the dumbing down of examinations. The beauty of an OT question is that the marking is completely accurate, no marking bias at all. This is often ignored in traditional exams and not seen as a problem largely because it’s not that visible. But the OT does not solve the “how much to test” problem, in fact it makes it worse. If you are asking for less then you are examining less. So if the only benefit is the avoidance of marker bias why are more examining bodies using OT style exams, is it just about saving money…….?

An example

Imagine that you have 4,500 OT style questions that cover every aspect of the syllabus, let’s also assume that the student only has to answer 50. The 50 questions are randomly picked from the 4,500 in the question bank. Is it fair that a student is considered competent if they are only being tested on 1% of what they need to know?

I think the answer is Yes, because in order to be sufficiently prepared to answer 50 questions from a bank of 4,500 when there are no core topics you have to have be capable of getting all 50 correct (assuming a 100% pass mark) and because you don’t know which 50 are coming up this effectively means you have to be able to get all 4,500 correct. The examining body can of course control  the effectiveness and level of difficulty by changing the pass mark.

All OTs would be OTT

This is not an argument to suggest that all examinations should be assessed using OT type questions, they should not. For example they are not very effective at measuring a student’s ability to communicate or evaluate complex and ambiguous situations, but they should be considered part of the tool kit that examining bodies have in assessing competence.

So on the face of it OT’s may look like a soft option, anyone can tick a box but they are certainly not easy to get right. Maybe less is actually more…..