Putting the context into case study

Context

I am still reading Sensemaking by Christian Madsbjerg and as I always tend to do I have been trying to reduce the 216 pages down to something that is both meaningful and memorable. The rational for this is that if I can summarise the essence of what is being said into a single statement, then my level of understanding is reasonably good, and it makes it easier for me to use what I have learned in other situations.

So here goes, if I was to summarise what Sensemaking is all about, in one word it would be..….Context. In essence, in a world of complexity and abundance of information we are in danger of thinking that the “fact” we see on our computer screen, offered up by a search engine, driven by an algorithm is the truth, when in reality it’s only one version of it. Without the context from which this information came we are fooling ourselves as to its true meaning.

As a result of this discovery, I wondered into an area I  have wanted to write about before, the importance context plays in changing what something means, especially in examinations. Getting the meaning wrong could be the reason you fail the exam rather than pass it.  Even objective tests will have some form of context setting just before the actual question. But the type of exam where you are most likely to have a problem with context, is a case study.

Jokes play with context

A hamburger and a french fry walk into a bar.

The bartender says, “I’m sorry we don’t serve food here

The importance of context in case study

I have written about case studies before, “passing case study by thinking in words,” but focussed more on the process of how you think and write rather than how you interpret the information presented.  Case studies are becoming an increasingly popular way of assessing a student’s ability to apply knowledge from several different subjects (synoptic) in the context of a real-life situation.  This shift towards case studies is understandable given the need for improved employability skills. Here is a great story to illustrate how context changes the decision you would make or as often in a case study, the advise you would give.

A battleship had been at sea on its routine manoeuvres under heavy weathers for days. The captain, who was worried about the deteriorating weather conditions, stayed on the bridge to keep an eye on all activities.

One night, the lookout on the bridge suddenly shouted, “Captain! A light, bearing on the starboard bow.”

“Is it stationary or moving astern?” the captain asked.

The lookout replied that it was stationary. This meant a collision would result unless something changed. The captain immediately ordered a signal to be sent to the other ship: “We are on a collision course. I advise you to change course 20 degrees east.”

Back came a response from the other ship: “advise you change your course 20 degrees west.”

Agitated by the arrogance of the response, the captain asked his signalman to shoot out another message: “I am the captain of one of the most powerful battleships in the British navy, you change course 20 degrees east now.”

Back came the second response: “I am a second-class seaman, you had still better change course 20 degrees west.”

The captain was furious this time! He shouted to the signalman to send back a final message: Change course 20 degrees east right now or you will leave me no choice!

Back came the flashing response: “I am a lighthouse – your move.”

How to deal with context

It is easy even in the example above to think you know what is going to happen or what you would do. But when the context is revealed, your advice fundamentally changes. Case studies are created to see how well you respond in certain situations, so it’s important not to jump to conclusions.

And this is where sensemaking plays its part, use your senses, don’t just look at what is there, think in opposites, what is not there, what’s missing? Use visualisation, see yourself in that situation, look around, free up your thoughts, what do you see now? But most of all, be curious, ask questions of the scenario, how big is the ship, how long has the captain been in charge, what is the weather like, are there others close by?

Another excellent tool to use in these situations is called perceptual positions. Think of the event from different positions, firstly yours, what does the event look like through your eyes, secondly, the other person(s), what would you do if you were them, and thirdly what would the event look like if someone was looking in, observing both parties.

Case studies in the future will become even more sophisticated. Virtual reality offers up so many opportunities to create real world environments in which to tests students. And when that happens, you will definitely need to use all of your senses to get you through – take a look at this 360 VR surgical training, amazing.

And one last joke

Thomas Edison walks into a bar and orders a beer.

The bartender says, “Okay, I’ll serve you a beer, just don’t get any ideas.”

 

 

 

 

 

 

Teaching to the test – another point of view

point-of-veiw-2A point of view is a programme on radio 4 that allows certain well-read, highly educated individuals, usually with large vocabularies to express an opinion. It lasts 10 minutes and is often thought provoking, concluding with a rhetorical question that has no answer.

This week Will Self the novelist and journalist gave his point of view on teaching to the test, as you might imagine it caught my attention. Self starts by telling a story about the life of a “good student,” and how it would unfold. He describes the way in which their concentration intensifies when the teacher states that what they are going to learn next is important and often examined. The story continues, as a result of their diligence and technique, the “good student” gets the necessary grades to go to University. They don’t however select the University on the basis of the course of study or on what they passionately wish to learn, no its based on the Universities credibility in league tables.

Upon successfully gaining a degree the student, now an employee gets a desk job that rewards a similar style of rubric mentality. As an employee, they are assessed against targets, performing well only on the ones that promise promotion and a pay rise. Eventually they retire and die.

Self concludes that this ordinary, dull, uninspiring life started back in the classroom all those years ago, when the teacher failed to educate and inspire, and simply taught to the test.

Over egging the pudding

There is a logic to this story, and it sounds ever the more inevitable as Self narrates it in his black and grey voice. But that’s all it is, a story. It avoids detail and colour, offering little regard as to the individual’s ability to reflect at some point in their life and ask searching and probing questions. It is as if somehow because the teacher highlighted the importance of one piece of knowledge it somehow stifled the student’s capacity to one day think for themselves.  Self is how they say, overegging the pudding, taking an interesting question as to the impact teaching to the test might have and serving up an omelette.

Teaching to the test is not bad

Brunel university asked a question as to what makes an unmissable lecture. In addition to many arguably more commendable answers, including the passion of the tutor and because they wanted to learn, the likelihood of the subject being taught having a high probability of being in the exam was key. Suggesting that a specific topic might be on the exam paper firstly, ensured a good attendance and secondly guaranteed the student listened intently.

Attention is important but even for the diligent student focus is vital. Learning everything is simply not possible, faced with 20 chapters, the student needs some clue as to where they need to direct their energy and time. Of course, the educationist will say that everything is important, but saying that will not make it so. Knowing that something is examinable at least gives a starting point and helps guide the student through the material quickly and efficiently. It’s also worth adding that It does not exclude the need to be inquisitive, in fact by making the student read a particular topic it may inspire them to find out more.

Exams and exam answers also provide examples of what is expected and the standard the student must reach if they are to be successful, no amount of narrative in the student handbook or curriculum guidance will do this as effectively.

The type of assessment matters

Of course, in Selfs world, teaching to the test removes the need to do anything more than learn about what will be in the exam. He suggests that students need to think outside the box rather than simply tick them. I have to admit I like that sentence.

But he does have a point, if the test is so narrow that it only assesses memory or a very small part of the syllabus then that is all the student will focus on. But that is just a bad test, this is of course where I am in danger of becoming idealistic and painting a picture that is not a true reflection as to what is happening. Not all tests are good, and undoubtedly some students will pass with limited thought and little more than good memory skills. Yet with changes in technology it becomes ever more possible to build tests and simulations that asses the student ability to perform in real world situations, and for that matter think outside the box.

Teaching to the test has become a term used to describe bad teaching and poor assessment and no one would agree that either of these are desirable. But it is not the process that’s problematic, it’s the application. Testing in its many forms is part of learning but it needs done well and thoughtfully.

In conclusion

Having now read the blog I would encourage you to listen to Will Self – click. It is of course not for me to say who presents the right point of view, you need to make up your own mind. For those however who were taught to the test no matter how long ago, you probably won’t understand even what I am asking because to the best of my knowledge this question has never been tested before……………?

 

The Paradox of on demand exams

paradox-of-choice-1

But which one is best when there is so much choice?

It is more blessed to give than receive, so it says in the bible, and Christmas is a time when you get to do this in abundance. What better feeling can there be when you find exactly the right present for that special person. You can even imagine the smile on their face when on Christmas day they rip open the paper to reveal the gift they either never new they wanted but did, or in their wildest dreams believed they could have or own.paradox-wonderful-christmas-time-1

BUT…….its not as easy as it used to be. Deciding on what you want to buy is one thing, finding the “best” or most suitable gift is another.

Paradox of choice

Although Psychologist Barry Schwartz wrote his book called The Paradox of Choice over 12 years ago, most if not all of what he said is more relevant today with the explosion of choice created by the internet and advances in technology. He argued that choice rather than being good, and by the way the basis for freedom, was not necessarily bad, but did not always result in the best decisions being made or to be precise, the feeling that the best decisions had been made.

There were two reasons for this, firstly, too much choice creates paralysis, a delay in the decision-making process (see also Buridan’s ass) that can result in no decision being made at all, and secondly if you do decide, ultimately you will become less satisfied with your choice for a variety of reasons e.g. regret, opportunity cost and the escalation of expectation.

But what has this all got to do with examinations?

On demand exams result in their own paradox

In the world of professional examinations for many years’ students and employers have been asking for more choice as to when the exam can be sat. Its only tradition and inflexible systems that have created a need for testing to take place at specific times of the year. But with technology comes flexibility and that flexibility now means that some examinations can be sat when you want, not when the professional body or university decides. Driving tests have long followed the concept of, “on-demand” testing.

This choice is however resulting in a paradox. Students are taking longer to pass their exams but despite having more time results are not improving. You would expect that having additional time to study would improve exam success not reduce it.

One of the reasons for taking longer to pass is because it is now within the student’s power to change the exam date if they so wish. There are a number of good reasons for doing this, for example, work pressure, not feeling ready for the exam or that you dont know enough. Logical reasons but are they “good” reasons?

That all depends of course on the objective, but if the objective is to pass the exam quickly, then the answer is no.

The best of both worlds

Schwartz never concluded that having a choice was bad, just that it was not the end game. Ultimate choice is not the objective. He did say that we should perhaps stop thinking in terms of maximising choice but set some standards that can be used to help navigate the choices available.  In fact, this is borne out by some initial research into this area which shows that students increase their chances of passing if they set an exam date, (the choice) enable sufficient time to study but don’t change the exam date later (the standard). The choice when to pass can be set but neither the length of study nor the exam date should change.

Alternatively:

Consult an expert – What do people do when they are faced with such a wide range of options they cannot decide, how about asking an expert? In this instance the expert would be the educational establishment or the examining body, what do they recommend?

Follow the norm – Data is now more readily available than ever before, many technologies make use of historical trends to make predictions or to offer advice when there are lots of choices. Everyone will be familiar with the web sites that suggest books or other items you should buy based on past behaviours. What worked best for you in the past, what did you do before that was successful or what do the most successful people do?

Now that we have cleared up how to make better decisions in a world of endless choices – I need to begin thinking about my New Year’s resolutions, oh dear if only there were not so many choices.

Happy new Year to you all

Click to watch Barry Schwartz TED lecture.

 

 

Currywurst , apples and the £33,000 a year boarding school

currywurst

Currywurst fast becoming the most popular dish in Berlin

I have three relatively unrelated stories this month.

Berlin

The first follows a trip to Berlin this weekend and the obligatory city tour. Just of Unter den Linden you will find a sunken glass plate between the pavement that provides a view into a room full of empty bookshelves, sufficient to house the 20,000 books that were destroyed on this site on the 10th of May 1933. Some 70,000 people, many German university students gathered to burn books with “unGerman” ideas. Joseph Goebbels joined the students at the bonfire and declared: ” The future German man will not just be a man of books, but a man of character.” “It is to this end that we want to educate you.”

This was not only symbolic but practical. Without these books or to be precise only having the ones that remained, it might just be possible to create a nation of people who all think the same? Who all share the same ideals and live life by the same values. And yet of course this is not how you create an educated society, a society able to think for itself, the result will be compliance and idealism. Which was more likely what Goebbels had in mind.

Goodwill Hunting

When I got home one of my favourite movies had already started. Good Will Hunting tells the story of a maths genius, Will Hunting who struggles to come to terms with his past and as a result is unable to make a commitment for fear of being let down. But for me the best scene is where Will confronts a first year grad student who has been belittling his friend Chuckie.

Will: “Wood drastically underestimates the impact of social distinctions predicated upon wealth, especially inherited wealth…” You got that from Vickers. “Work in Essex County,” Page 98, right? Yeah I read that too. Were you gonna plagiarize the whole thing for us, you have any thoughts of your own on this matter? Or is that your thing, you come into a bar, you read some obscure passage and then you pretend, you pawn it off as your own, your own idea just to impress some girls? Embarrass my friend?

Will: See the sad thing about a guy like you, is in about 50 years you’re gonna start doin’ some thinkin’ on your own and you’re gonna come up with the fact that there are two certainties in life. One, don’t do that. And two, you dropped a hundred and fifty grand on a fxxxin’ education you coulda’ got for a dollar fifty in late charges at the Public Library.

Clark: Yeah, but I will have a degree, and you’ll be serving my kids fries at a drive-thru on our way to a skiing trip. Will: [smiles] Yeah, maybe. But at least I won’t be unoriginal.

This of course challenges the idea that knowledge is the same as thinking. Knowing something is not the same as having manipulated the information in your head, questioned and explored the arguments from many angles, the result should be an opinion, not someone else’s but your own.

Wellington college

And finally to bring my trilogy up to date. Only this week Wellington College who boast George Orwell and Sebastian Faulks as past students announced that have introduced a six-hour interview process to help teachers see beyond children’s over tutoring. They encourage families to spend more quality time instead of having their children tutored, Julian Thomas, Wellington’s head master, said: “When parents ask how should they prepare their children for our selection process, we tell them ‘have supper together and talk as a family; go to plays and good films and discuss them. Help them to think critically about the world around them; enjoy interesting conversations go out for walks and see the world”.

What Wellington are trying to do is encourage behaviours that will help children become thinking adults.

In conclusion

Knowledge like information is becoming increasingly accessible, unlike Germany in 1933 we have no restrictions, the internet has given us that power. But learning is more than knowledge you have to take the facts and make them your own, stitching them together carefully and thoughtfully, and that takes time and a considerable amount of effort. On the whole absorbing information will probably not come as easy to you as it did to Will Hunting, yet even with his genius he only knew what he had read. His learning had in fact only just begun as he went looking for the real life experiences that would test if he really understood anything at all.

Examinations are moving more towards tests that require you to think, asking that you provide an opinion or make recommendations given a set of circumstances, case studies are a good example. Of course you need knowledge as a foundation but the ability to sift quickly through the vast amount of information available may in time become more important than knowledge itself .

Ps apples is taken from Good Will Hunting – “How ya like those apples”

 

 

The tip of the iceberg – exam tipping is becoming obsolete

tip1

Assessment is changing, there was a time when all examinations were sat in a room, the answers would be hand written on a piece of paper and a retired English teacher would stand at the front reading out instructions as to what you could and couldn’t do in the next three hours.

Not any more…….you request a date that is convenient, turn up at the exam center, no longer is this a sports hall, it might be a driving school test center or the college you studied at. Then you log onto the PC and answer questions on the computer screen in front of you. The results may be immediate; it depends on if it is “human marked” or computer marked.

But in some ways these changes are only the tip of the iceberg!

What no past exam papers.

As examinations move into the digital world we are seeing other changes as well. There is a move towards objective testing, scaled scoring and examining bodies no longer providing past exam papers, what did you say, no past exam papers……!

This is partly down to the nature of the test i.e. you can’t provide an exact replica of a past exam question if it is an objective test. Remember objective test questions are randomly selected from a pool, and are different for each student. But there is also a shift towards some examining bodies only providing an example of the type of questions that could be set rather that a continuous flow of, the last exam papers.

If the test changes – how you study (and teach) has to change

Now for someone who has advocated that students analyse past exam questions in order to identify key areas so as to better direct their studies, this is a bit of a blow. It has also been the method I have used in the past to focus my own delivery in class and on line. Of course using past exam questions has always been much more than just spotting key areas, it is about focus, providing a place to start, showing content in the right context, helping with writing style etc.

There will still be past questions, sample questions will be provided. What we don’t know is how representative they will be of the examination. Or will it be as we have seen in the past with pilot and specimen papers, they change over time, drifting away from the original in terms of style and emphasis. Although I can see the logic in examining bodies not releasing papers, I hope they will continue to keep the sample papers fresh, in keeping with current thinking about the subject and how it will be examined.

What to do?

Students and tutors still need focus, there has to be emphases on key areas in order to chunk the content so that it can be more easily learned, it’s just that we won’t be able to use past questions or at least as much as we have in the past. That emphasis will now have to come from articles written by the examiners, examiner reports and syllabus weightings. If faced with a new subject where there is only one sample paper, it will be necessary to read the guidance from the body closely, noting reference to “this being a key part of the subject” or “one the examiner thought was answered badly in the past.” These together with the syllabus weightings and specific learning outcomes will have to be your guide. It is of course possible that the subject has not changed much from before and so some of the older past question can be used. As far as questions style is concerned then that will have to come from the questions and answers that are published, it may not be ideal but it’s the best we can do.

The overall impact of these changes is that students will have to know more, something that is hard to argue with. Students and tutors alike will have to devote far more time to the subject, which is fine if students have the time and can afford the extra costs involved in longer periods of study.

But it’s not all bad news, new technologies can help students make the most of dead time, studying on the train using their mobile phone for example. Also knowledge is more freely available than ever before as many top institutions provide a huge amount of free easy to access content online.

One final thought, examinations may change and they may not be fair but on the whole they are equal, everyone as before is in the same boat, and someone will always pass, wont they!

The future – Sitting the exam at home?

On line exams

An online student, all be it a mature one shows his ID to the online assessor

And maybe even the exam room will become obsolete. Proctur U is a US based company that also has a presence in the UK offering online invigilation. Watch this video to see how it works and judge for yourself

 

 

So you want to be an astronaut – assessment for astronauts

 

Astronaut-Tim-Peake-Dials-Wrong-Phone-Number-from-Space

Your there by yourself. There’s no doctor, there’s no computer engineer – so you have to learn all of these skills. Tim Peake

What is the purpose of assessment?

In order to gain a better understanding of the assessment process for astronauts, let’s ask a more basic question first, what is an assessment and what does it prove?

At one level it is simply a measurement of performance benchmarked against a given outcome or standard. The results can, help a teacher identify progress so they can adapt the next lesson, or give assurance an individual is capable of performing a particular task. This is probably the most important type of assessment for astronauts.   Modern high stakes “examinations however play another role; they offer the student a transferable badge of honour that can help open doors to better career prospects and increased salary. If you doubt the importance of this last point, click this link to read about the level of corruption in India and the lengths people will go in order to obtain a certificate saying they have passed an exam.

Employability gap – so what’s the problem?

However examinations on their own do not provide a guarantee that the individual who passes will be able to do the job.  And it is here where the disconnect becomes clear. Often the method of assessment is not based on what the student will be doing in the work place, and even if it is, it is not set in the same context. Many believe this gap has become ever wider as more and more students come out of school/university lacking the skills required by the employer.

But should the exam/assessment be changed to narrow the gap or is it the role of the employer to provide the necessary “on the job” education in the work place? However the problem with the employer being given the responsibility, it implies that what you study doesn’t matter, only that you do. This just seems wasteful, wasteful of time, money and effort. Assessment must get closer to measuring the skills required for one simple reason, what gets measured gets achieved, a cliché for sure but a true one.

Assessment for astronauts

Costs – The costs involved in training astronauts is a bit unclear, NASA will pay Russia $70m for a seat on Russian space craft in 2017 that includes training, figures of around $25m are also cited, Tim Peaks training was quoted as £16m so very similar.

Basic requirements – Mission Specialist (non pilots – most are pilots with a military background) include the following:

  1. Bachelor’s degree from an accredited institution in engineering, biological science, physical science, or mathematics. The degree must be followed by at least three years of related, progressively responsible, professional experience.
  2. Ability to pass a NASA space physical, which is similar to a military or civilian flight physical.
  3. Height between 58.5 and 76 inches.

Training – Tim Peak’s education started in 2008 when he was selected by ESA (European space agency) from 8,413 applicants. Tests at this stage included, intelligence, spatial awareness and concentration. Then for the 10 selected another 18 months of intense training followed, this time on topics as wide ranging as space law, rocket propulsion, spaceflight engineering and the hardest for Tim, learning to speak Russian. He was also subject to one week’s caving in Sardinia, Italy, with five other astronauts to simulate what it would be like in space, this was to build teamwork, problem solving and cope with poor hygiene facilities. Then it was one year of advanced training, including working underwater to simulate the lack of gravity. In 2013 it was time to go into space for the first time, this was the last test before being selected and then finally in 2015 Tim was allowed to do the job for real. In all it had taken 6/7 years.

What have we learned?

Clearly you need some underpinning academic skills i.e. a degree but a relevant one. The training is provided by the state, the argument being that it is for the greater good of society so the costs should be met by the tax payer. An interesting point for those that believe this should be the case for all education. But most importantly the assessments were all built around making sure that when Tim got into the space he could do the job. Very few of the tests required a piece of paper and a desk, and many were simulations of what he would meet in the real world.

Given the advances in technology the time is now right to introduce more simulations into the exam room, not so we can all become astronauts but to help prepare the next generation for the work place.

And just imagine your badge of honour when going for that next job – “what exams have you passed” “oh a few, but did I mention I was an astronaut…..”

 

The mystery of scaled scoring

A mystery is defined as something that is difficult to understand and or explain. This describes exactly the situation we have at the moment as many examining bodies move their assessment towards scaled scoring and away from the more traditional percentage pass mark.  Some argue that scaled scoring is clearer in that it allows direct comparison of one examination to another but from my experience in the short run at least clarity is not the term many would use.

The raw score

How did I get on?For most students the way to find out how well they are doing is to have a piece of work marked. If the total marks available are 50 and the student scores 35 then as a percentage they have obtained 70% of the available marks. This is called the raw score. Interestingly it tells you very little, for example is 70% good or bad? For a raw score to have any meaning it needs to be compared with a standard. This could be with another student or group of students, and is an example of what is called norm referencing, now this clearly has meaning as you can tell if you are performing better or worse than your peer group. But the most common way in which exam results have historically been published is as a pass/fail against a pass mark or as it is sometimes called a cut score. If the pass mark is 60% and the student has scored 70% then they have passed.

The scaled score

However raw scores are only fair if the exam sat is the same for every student. It would be unfair to give each student a different exam and set the same pass mark, unless of course you believed the exam was of an equal standard.  This has been the argument in the past, yet producing an exam of equal standard is almost impossible, and this is why scaled scores are being used more often.

To use scaled scoring you need a range of marks, for our purposes let us assume the range or scale goes from 0 to 150. We then have to set a common passing score, say 100, this is the score on the scale that the student needs to achieve to pass the exam or test, and it will never change. What happens next is what makes a scaled score so useful.

ExpertsA group of experts including the examiner review the exam that has just been sat, they will probably mark a number of scripts themselves and will have access to the results from all other students. Then taking into account the performance of all students and their knowledge of this particular exam they will set the pass mark for this paper. Let’s assume that the pass mark is 32, if a student is awarded a raw score of 32 on this particular paper it will be equated to a scaled score of 100. All other scores above and below 32 will also be given a scaled score equivalent.

So on some levels using a scaled score is no different to the examiner setting a different pass mark for each exam, acknowledging that the level of difficulty will never be the same for any one examination. The reason the scaling is used however is because it can be stated that the passing score for the exam is always 100, this means the standard stays the same even though the marks may change.

So there you have it, no longer a mystery….I hope?

For more information and guidance watch the video on Pearson’s web site. Click on this link and scroll down.

 

 

 

 

 

 

 

 

 

Playing the game – Tips for answering objective tests

Amazon

Which is the real logo?

Objective tests (OT’s) are becoming increasingly popular.

There are many reasons for this; educationally one of the big advantages is because, they are objective i.e. the mark is accurate, it’s either right or wrong.

It may come as a surprise to some students but when people mark there will be a slight (we hope) difference, its called marker bias.

The more cynical might argue that the examining body introduces them to simply save money. Whatever the justification students will find OT’s being used more frequently, at all levels and so need to be prepared.

But OT’s must be easier?

Answering a question like…. which one of the following best explains how a car engine works a-b-c-d, must be easier than writing/typing an answer explaining how a car engine works – right?

Well perhaps not, what if all of the answers are plausible? Think of the mental process you go through trying to distinguish between them, digging deep into your understating, looking for a word in the question that might give a clue, might help you narrow down your options. And even if you do narrow them down to let us say, 50:50, there are no method marks, it’s either right or wrong; it’s 50% of a full mark or 50% of no mark. The student answering the written question is unlikely to get no marks at all even on a question they don’t really understand.

Personally I think OT questions are more difficult from a student’s perspective but very useful for examining bodies as part of the assessment process. But they should only be part of that assessment process; other types of assessment should also be used.

Playing the OT game – Tips

But what can be done, how can you improve your chances of passing OT’s?

Well there are the simple things like, make sure you read the question carefully. This is much easier if it is a paper based test where you can underline exactly what you have been asked to do. I have written in the past about the importance of underlining. It helps the brain focus on what is important and what is not.  This is made much harder when the OT questions are on the screen. In these circumstances I would suggest you write out the key words on a pad or white board.multiple-choice marriage'

Answer the questions you can answer first and leave the longer more debatable questions until the end, and follow the advice of Ludy T. Benjamin, et. al (1984). She identified you are better changing your original answer to another one if you doubt it. This is very much the opposite of conventional wisdom that suggests the first answer you come up with is probably correct. But be careful, this is only if you doubt your original answer. The argument being that when looking at the question a second time you can tell something isn’t right and so will spend more time on the question than before, changing to a more plausible one.

There are some more sophisticated techniques that can help reduce the odds, I have summarised them below. Many of these I have borrowed from a more comprehensive article written by a colleague, John Bennett – thanks John.

  • Distracters – these are questions that contain an answer very similar to the real one and are often plausible. The technique is to cover up all the answers, so that they don’t distract, work out what you think the answer is, then reveal. Hopefully your answer will be in the list.
  • Go for the long answers – William Poundstone author of “Rock Breaks Scissors” noticed that the longest answer on multiple choice tests was usually correct. “Examiners have to make sure that right answers are indisputably right,” he says. “Often this demands some qualifying language. They may not try so hard with wrong answers.”
  • Eliminate the outliersAnother Poundstone tip is to look out for one of the answers that is very different to the others, and if you find the outlier it’s probably wrong. So for example if you had 4 numerical answers a£0.46 – b£0.54 – c£0.55 – d£1.60. The outlier in this sequence of numbers is d and is unlikely to be correct. It doesn’t of course give you the answer but it will at least improve your odds.
  • Find opposites – An easy one next, where two answers are exact opposites, the answer is more likely to be one of those two.
  • Look out for general words – The University of Minnesota identified that the question that use general words such as, mostly, possibly, often, usually, will “often” be the correct one.  This is because when an examiner wants to write an incorrect answer they will be far more specific e.g. it will NEVER rain on Friday as opposed to Friday is OFTEN the wettest day of the week.
  • Negative worded questionswhen questions ask which of the following is NOT true mark off the ones that ARE true first. The brain struggles to recognise negatives, so you need to put the question in terms of positives, what it is as appose to what it is not.

There are of course more techniques but you don’t want to enter the exam worrying more about the techniques than the exam itself.

And don’t forget hard work – exam tips only stop you failing

And as you would expect me to say, these tips only help you play the game better, hard work, studying and practising questions are far more important.

The answer is 3 – Amazon sells everything from A – Z and that puts a smile on your face

Are exams fit for purpose (part two) – what are the alternatives?

You dont fatten pigs 2

Last month’s blog came to the conclusion that examinations* are fit for purpose or at least “a purpose.”

They provide the student with a clear objective to which they can direct their efforts and focus attention and are a transferable measure of competency that can be assessed at scale. The “at scale” point is important as there are many ways of assessing competence but few that can cope with the need to test thousands of students all at the same time.

The main problem with examinations is that they don’t always examine what is most valued; the method of assessment often has significant limitations as to what it actually tests and the results are presented in league tables that give a far too simplistic view of success.

I am not sure we can resolve all of these but it might be worth exploring other options, specifically alternative methods of assessment. For example If you change the method of assessment from a formal, often timed written exam to say a portfolio of work, not only do you change the method of assessment but you will change what is being examined, two birds with one stone perhaps.

Different methods of assessing competence

Open book exams

Open book assessment offers a way of testing application rather than memory. Students have access to a text book that contains information relevant to what they are being asked. It’s the use of knowledge that is important, not the knowledge itself. The idea of open book could easily be adapted, why not allow students access to the internet during the exam, they could look up anything they wanted. Is this not more representative of what happens in the real world?

Take out exams

Similar to the above the so called “take out exam” allows the student to take the exam away to work on at home using whatever resources they prefer, books, internet etc. They return the next day with a completed answer. This can work better than you might at first think so long as you have a robust mechanism to detect plagiarism. There are several very good software packages that can spot the most sophisticated types of copying.

Case studies/simulations

A case study provides an environment for the student to demonstrate they can use their knowledge to solve problems and or offer advice in a virtual world. Most case studies tend to be written but this is one area that we could see some clever and affordable use of technology to better simulate the real world.

Performance tests

In a performance test students are required to demonstrate a skill/process, create a product etc while being observed by the assessor who will evaluate the performance. A great example of testing ability to apply knowledge but suffers from the subjectivity of the assessor and has limited application at scale.

Portfolios

Portfolios are most often collections of the student’s work that demonstrate their ability to perform a specific task. These can be simulations of the real world or portfolios of work actually undertaken on the job. A portfolio can include written documents, emails, audio or video recordings, in fact anything that provides evidence as required by the assessor.   Portfolios are perfect for assessing application but the process of assessment is expensive and not without bias.

Viva Vocal – (living voice) Oral exam

Often used to test PhD students, an oral exam gives the assessor chance to question the student. This is a very effective method where you are looking for higher level skill and depth of understanding. As identified last month it’s probably one of the oldest forms of assessment.

Digital badges – capturing the learning path

Being awarded a badge as recognition of achievement is something many will be familiar with especially if you were a boy scout or girl guide. But digital badging is new and becoming increasingly popular because of the internet. A good example would be linkedin and the badges awarded to you by others as recognition of certain skills.  Many of the assessment methods above provide a first past the post type of assessment, you pass and that’s it. Digital badging on the other hand is a form of lifelong assessment that evolves along with your career.

Digital badging for me is one of the most exiting forms of assessment and I am not alone Nasa have been using digital badging since 2011. Read more about digital badging.

Assessment in the future

Scanning for competenceThe list above is far from comprehensive and many other equally valid types of assessment exist e.g.  Role plays, Slide presentations, Assignments etc but what might assessment look like 15 years from now. Well how about using MRI scans to identify which parts of the brain are being used?  Not sure it will catch on but it would provide some interesting evidence as to how the student is getting to the answer, simple memory or a genuine and deep understanding .

*Examinations defined as a written test administered to assess someone’s level of understanding, knowledge or skills

50 Shades Darker – How much to test

How much to test

In August last year I wrote how Professional accountancy bodies believe that competency can be measured by a candidate scoring 50% and failure if scoring 49%. This all or nothing approach seems neither fair nor useful; hence the idea that grades of competence could be introduced e.g. 50% pass, 65% commendation, 75% distinction etc.

But the mark a student gets in an exam is only part of the story when it comes to measuring competence. Can a student be considered competent if the exam they pass only includes questions from say 75% of the syllabus? Yes the whole syllabus might be covered in an 18 month period but in any one exam 25% is not tested. Equally the 75% is often considered core and so examined every sitting, this means that a student need only focus on the 75%. Admittedly if the pass mark is 50% they need to score 50% out of 75% (67%) but with practice this is possible. One final observation, it is unlikely the student will score 0% on the non-core part of the syllabus. They may get say 5% or even 10% out of 25%. Not a great score but the 67% pass mark now becomes 58%. This logic sits at the heart of the exam driven approach.

Objective testing might be the answer?

Objective tests (OT) – test that are relatively short and can be unambiguously marked, are considered by some to be a weaker form of assessment, they are part of the dumbing down of examinations. The beauty of an OT question is that the marking is completely accurate, no marking bias at all. This is often ignored in traditional exams and not seen as a problem largely because it’s not that visible. But the OT does not solve the “how much to test” problem, in fact it makes it worse. If you are asking for less then you are examining less. So if the only benefit is the avoidance of marker bias why are more examining bodies using OT style exams, is it just about saving money…….?

An example

Imagine that you have 4,500 OT style questions that cover every aspect of the syllabus, let’s also assume that the student only has to answer 50. The 50 questions are randomly picked from the 4,500 in the question bank. Is it fair that a student is considered competent if they are only being tested on 1% of what they need to know?

I think the answer is Yes, because in order to be sufficiently prepared to answer 50 questions from a bank of 4,500 when there are no core topics you have to have be capable of getting all 50 correct (assuming a 100% pass mark) and because you don’t know which 50 are coming up this effectively means you have to be able to get all 4,500 correct. The examining body can of course control  the effectiveness and level of difficulty by changing the pass mark.

All OTs would be OTT

This is not an argument to suggest that all examinations should be assessed using OT type questions, they should not. For example they are not Less-is-Morevery effective at measuring a student’s ability to communicate or evaluate complex and ambiguous situations, but they should be considered part of the tool kit that examining bodies have in assessing competence.

So on the face of it OT’s may look like a soft option, anyone can tick a box but they are certainly not easy to get right. Maybe less is actually more…..