The Paradox of on demand exams


But which one is best when there is so much choice?

It is more blessed to give than receive, so it says in the bible, and Christmas is a time when you get to do this in abundance. What better feeling can there be when you find exactly the right present for that special person. You can even imagine the smile on their face when on Christmas day they rip open the paper to reveal the gift they either never new they wanted but did, or in their wildest dreams believed they could have or own.paradox-wonderful-christmas-time-1

BUT…….its not as easy as it used to be. Deciding on what you want to buy is one thing, finding the “best” or most suitable gift is another.

Paradox of choice

Although Psychologist Barry Schwartz wrote his book called The Paradox of Choice over 12 years ago, most if not all of what he said is more relevant today with the explosion of choice created by the internet and advances in technology. He argued that choice rather than being good, and by the way the basis for freedom, was not necessarily bad, but did not always result in the best decisions being made or to be precise, the feeling that the best decisions had been made.

There were two reasons for this, firstly, too much choice creates paralysis, a delay in the decision-making process (see also Buridan’s ass) that can result in no decision being made at all, and secondly if you do decide, ultimately you will become less satisfied with your choice for a variety of reasons e.g. regret, opportunity cost and the escalation of expectation.

But what has this all got to do with examinations?

On demand exams result in their own paradox

In the world of professional examinations for many years’ students and employers have been asking for more choice as to when the exam can be sat. Its only tradition and inflexible systems that have created a need for testing to take place at specific times of the year. But with technology comes flexibility and that flexibility now means that some examinations can be sat when you want, not when the professional body or university decides. Driving tests have long followed the concept of, “on-demand” testing.

This choice is however resulting in a paradox. Students are taking longer to pass their exams but despite having more time results are not improving. You would expect that having additional time to study would improve exam success not reduce it.

One of the reasons for taking longer to pass is because it is now within the student’s power to change the exam date if they so wish. There are a number of good reasons for doing this, for example, work pressure, not feeling ready for the exam or that you dont know enough. Logical reasons but are they “good” reasons?

That all depends of course on the objective, but if the objective is to pass the exam quickly, then the answer is no.

The best of both worlds

Schwartz never concluded that having a choice was bad, just that it was not the end game. Ultimate choice is not the objective. He did say that we should perhaps stop thinking in terms of maximising choice but set some standards that can be used to help navigate the choices available.  In fact, this is borne out by some initial research into this area which shows that students increase their chances of passing if they set an exam date, (the choice) enable sufficient time to study but don’t change the exam date later (the standard). The choice when to pass can be set but neither the length of study nor the exam date should change.


Consult an expert – What do people do when they are faced with such a wide range of options they cannot decide, how about asking an expert? In this instance the expert would be the educational establishment or the examining body, what do they recommend?

Follow the norm – Data is now more readily available than ever before, many technologies make use of historical trends to make predictions or to offer advice when there are lots of choices. Everyone will be familiar with the web sites that suggest books or other items you should buy based on past behaviours. What worked best for you in the past, what did you do before that was successful or what do the most successful people do?

Now that we have cleared up how to make better decisions in a world of endless choices – I need to begin thinking about my New Year’s resolutions, oh dear if only there were not so many choices.

Happy new Year to you all

Click to watch Barry Schwartz TED lecture.



Playing the game – Tips for answering objective tests


Which is the real logo?

Objective tests (OT’s) are becoming increasingly popular.

There are many reasons for this; educationally one of the big advantages is because, they are objective i.e. the mark is accurate, it’s either right or wrong.

It may come as a surprise to some students but when people mark there will be a slight (we hope) difference, its called marker bias.

The more cynical might argue that the examining body introduces them to simply save money. Whatever the justification students will find OT’s being used more frequently, at all levels and so need to be prepared.

But OT’s must be easier?

Answering a question like…. which one of the following best explains how a car engine works a-b-c-d, must be easier than writing/typing an answer explaining how a car engine works – right?

Well perhaps not, what if all of the answers are plausible? Think of the mental process you go through trying to distinguish between them, digging deep into your understating, looking for a word in the question that might give a clue, might help you narrow down your options. And even if you do narrow them down to let us say, 50:50, there are no method marks, it’s either right or wrong; it’s 50% of a full mark or 50% of no mark. The student answering the written question is unlikely to get no marks at all even on a question they don’t really understand.

Personally I think OT questions are more difficult from a student’s perspective but very useful for examining bodies as part of the assessment process. But they should only be part of that assessment process; other types of assessment should also be used.

Playing the OT game – Tips

But what can be done, how can you improve your chances of passing OT’s?

Well there are the simple things like, make sure you read the question carefully. This is much easier if it is a paper based test where you can underline exactly what you have been asked to do. I have written in the past about the importance of underlining. It helps the brain focus on what is important and what is not.  This is made much harder when the OT questions are on the screen. In these circumstances I would suggest you write out the key words on a pad or white board.multiple-choice marriage'

Answer the questions you can answer first and leave the longer more debatable questions until the end, and follow the advice of Ludy T. Benjamin, et. al (1984). She identified you are better changing your original answer to another one if you doubt it. This is very much the opposite of conventional wisdom that suggests the first answer you come up with is probably correct. But be careful, this is only if you doubt your original answer. The argument being that when looking at the question a second time you can tell something isn’t right and so will spend more time on the question than before, changing to a more plausible one.

There are some more sophisticated techniques that can help reduce the odds, I have summarised them below. Many of these I have borrowed from a more comprehensive article written by a colleague, John Bennett – thanks John.

  • Distracters – these are questions that contain an answer very similar to the real one and are often plausible. The technique is to cover up all the answers, so that they don’t distract, work out what you think the answer is, then reveal. Hopefully your answer will be in the list.
  • Go for the long answers – William Poundstone author of “Rock Breaks Scissors” noticed that the longest answer on multiple choice tests was usually correct. “Examiners have to make sure that right answers are indisputably right,” he says. “Often this demands some qualifying language. They may not try so hard with wrong answers.”
  • Eliminate the outliersAnother Poundstone tip is to look out for one of the answers that is very different to the others, and if you find the outlier it’s probably wrong. So for example if you had 4 numerical answers a£0.46 – b£0.54 – c£0.55 – d£1.60. The outlier in this sequence of numbers is d and is unlikely to be correct. It doesn’t of course give you the answer but it will at least improve your odds.
  • Find opposites – An easy one next, where two answers are exact opposites, the answer is more likely to be one of those two.
  • Look out for general words – The University of Minnesota identified that the question that use general words such as, mostly, possibly, often, usually, will “often” be the correct one.  This is because when an examiner wants to write an incorrect answer they will be far more specific e.g. it will NEVER rain on Friday as opposed to Friday is OFTEN the wettest day of the week.
  • Negative worded questionswhen questions ask which of the following is NOT true mark off the ones that ARE true first. The brain struggles to recognise negatives, so you need to put the question in terms of positives, what it is as appose to what it is not.

There are of course more techniques but you don’t want to enter the exam worrying more about the techniques than the exam itself.

And don’t forget hard work – exam tips only stop you failing

And as you would expect me to say, these tips only help you play the game better, hard work, studying and practising questions are far more important.

The answer is 3 – Amazon sells everything from A – Z and that puts a smile on your face