50 Shades Darker – How much to test

How much to test

In August last year I wrote how Professional accountancy bodies believe that competency can be measured by a candidate scoring 50% and failure if scoring 49%. This all or nothing approach seems neither fair nor useful; hence the idea that grades of competence could be introduced e.g. 50% pass, 65% commendation, 75% distinction etc.

But the mark a student gets in an exam is only part of the story when it comes to measuring competence. Can a student be considered competent if the exam they pass only includes questions from say 75% of the syllabus? Yes the whole syllabus might be covered in an 18 month period but in any one exam 25% is not tested. Equally the 75% is often considered core and so examined every sitting, this means that a student need only focus on the 75%. Admittedly if the pass mark is 50% they need to score 50% out of 75% (67%) but with practice this is possible. One final observation, it is unlikely the student will score 0% on the non-core part of the syllabus. They may get say 5% or even 10% out of 25%. Not a great score but the 67% pass mark now becomes 58%. This logic sits at the heart of the exam driven approach.

Objective testing might be the answer?

Objective tests (OT) – test that are relatively short and can be unambiguously marked, are considered by some to be a weaker form of assessment, they are part of the dumbing down of examinations. The beauty of an OT question is that the marking is completely accurate, no marking bias at all. This is often ignored in traditional exams and not seen as a problem largely because it’s not that visible. But the OT does not solve the “how much to test” problem, in fact it makes it worse. If you are asking for less then you are examining less. So if the only benefit is the avoidance of marker bias why are more examining bodies using OT style exams, is it just about saving money…….?

An example

Imagine that you have 4,500 OT style questions that cover every aspect of the syllabus, let’s also assume that the student only has to answer 50. The 50 questions are randomly picked from the 4,500 in the question bank. Is it fair that a student is considered competent if they are only being tested on 1% of what they need to know?

I think the answer is Yes, because in order to be sufficiently prepared to answer 50 questions from a bank of 4,500 when there are no core topics you have to have be capable of getting all 50 correct (assuming a 100% pass mark) and because you don’t know which 50 are coming up this effectively means you have to be able to get all 4,500 correct. The examining body can of course control  the effectiveness and level of difficulty by changing the pass mark.

All OTs would be OTT

This is not an argument to suggest that all examinations should be assessed using OT type questions, they should not. For example they are not Less-is-Morevery effective at measuring a student’s ability to communicate or evaluate complex and ambiguous situations, but they should be considered part of the tool kit that examining bodies have in assessing competence.

So on the face of it OT’s may look like a soft option, anyone can tick a box but they are certainly not easy to get right. Maybe less is actually more…..

Thinking in box’s – Cracking case study

Put it in a boxExaminations come in many shapes and sizes, short form, multiple choice, essay, case study etc. I know there are other methods of assessment but I am thinking here of the most common. Of these one stands out as being very different, the case study. Developed by Harvard in the 1920s the  case study involves giving the student a  real life, normally business situation and asking relatively broad Socratic type questions e.g. what do you think, why etc.

It not only puts the student in a realistic situation but also requires them to think far more deeply.  The cognitive process involved in answering a question such as what is the capital of France or can you add 2+2, on the whole is very simple and may need little more than memory. However giving student a real life business to analyse and asking them to give an opinion as to what the company should do next requires higher level thinking as well as effective communication skills.

Too much to read

One other aspect of a case study is that it often involves large amounts of narrative, all will need reading digesting and putting into context. On the face of it this can seem daunting, but it can be done and as with so many aspects of learning there is a process that can help. I have written about, having too much to read and the benefits of chunking before but I want to bring these together with another powerful technique “thinking in box’s”.

Thinking in box’s 

Volume and lack of direction is the main problem here, so we need to find our own direction and reduce the volume. Thinking in box’s refers to the natural process we have in compartmentalising thoughts. In order to make sense of the world we often put “stuff” into box’s, work, study, relationships etc. We can then open the box’s when we are best able to deal with them. The  point being that  we can’t deal with everything all at once. If this whole idea sounds a bit odd, then just consider the saying “Thinking outside the box”. This refers to the imaginary frame we put around something that restricts our ability to solve a problem and think more creatively. Strange isn’t it…..

Case study

Imagine that you have 10 pages of narrative to read based on a particular industry, a case study. There are a few sub headings and some paragraphing. You are required to provide guidance to the board of Directors as to what the companies strategy should be in the next five years.

The process 

In order to give the advice required by this question, you need to fully grasp the current situation, which means you have to  read, understand and comprehend what is written on the 10 pages. To add structure to the case firstly take chunks of content and put a frame around it, this will help focus just on this chunk of information, it also reduces the volume. A chunk will often be information under a heading or specific paragraphs. Once you have the content in a box, sift through it looking for the “key words” and underline them.  Focusing only on the key words but taking into account the context, ask yourself, what do I think about this? What does it mean, what is it telling me etc. Then write down your thoughts. Do this for every chunk of information, then number each chunk.

At the end of this process you should have read and thought about each chunk, captured those thoughts and have a numerical reference by which to structure them. The final part of the process is to read each of those chunks again and produce a SWOT. This brings all 10 pages down to just one. And by using the SWOT supported by your detailed analysis you should be able to give the advice required by the question.

watch_this_videoIn this short video I demonstrate the thinking in box’s approach.

      Thinking in box’s

And finally – A few words from Terry Pratchet

I will be more enthusiastic about thinking outside the box when there is some evidence of thinking inside it!

50 shades of Grade…..Measuring students worth

I don’t often write specifically about the market that I am most closely involved, that of accountancy training and education. But there does seem to be an anomaly in the way the accountancy world measure success that isn’t the case in many other professions and examinations.

To become a qualified accountant in the UK (ACCA, ICAEW,CIMA etc,  yes there are more) you have to pass a number of demanding examinations and submit evidence of practical experience. The exams are taken over 3 years and everything rests on a number of exams of 100 marks each, of which you need to get 50%. The pass rate for accountancy students varies from paper to paper but is around 60% In training and education terms these exams are often referred to as High stake exams. For some failure is not simply a setback in terms of time, it could result in a lost job.

50% good – 49% bad

The purpose of such exams (including the practical experience) is of course to ensure that those that pass are competent. But how competent….?  It seems that you are either competent or not, if you get 49% you are not competent, if you get 50% then you are. This makes the margin for error very high, many a career has taken a dramatic turn based on 1 mark. Now of course in any exam there has to be a point where someone succeeds and another fails but does competence not have many shades to it, is it not a continuous process rather than a discreet one.

Shades of Grade

You may be one of thousands of students (>300,000 actually) who have just had your A level results. But of course you don’t just pass or fail, in fact this has not been the case since 1963, even before then there was an indication of the mark. You are given a grade from A through to E and all are considered a pass, 98.1% passed their A levels this year.

And the grades refer to the marks e.g. E is 40%, a D is 50%, a C is 60%, a B is 70% and an A is 80%. A similar grading exists for Degrees, 2.1, 2.2 etc and for Law exams, Pass, Commendation, Distinction.

No pressure for excellence

So why is this not the case in accountancy? To be honest I am not sure. But by introducing grades above a pass you not only provide employers with a better indication of competency, you encourage colleges to teach to a higher and deeper level, motivated by the students who want to learn more. And that for me is the most important point; you create an incentive for students to try harder. At the moment if you are scoring 55% then why put in more work, a pass is a pass, but a distinction well that will make you stand out from the crowd.

Of course the market will adjust its perception of what a pass means, for example employers may only want students with a distinction rather than a simple pass, but they have always looked and will continue to look for differentiation, at least this is a meaningful one.

What it does not do is fix the 49% you fail problem, and careers will still hinge on 1 mark, but that unfortunately is the way with exams and the reason exam technique is so important.

So to all the accountancy examining bodies why not introduce grades – grey is good and isn’t it an accountants favourite colour?