Category Archives: Assessment

Measuring success – “Authentic” Assessment

Posted on by

There is a saying often attributed to Peter Drucker – “what gets measured gets achieved.” I think the quote is “gets done” but that always feels a little clumsy. Although it’s easy to argue that the statement is crude and simplistic, it has a real-world truth.

Such is the inherent challenge with exams. Once a syllabus is written and a pass mark set, the students’ goal often shifts from trying to learn the subject to simply figuring out what needs to be done in order to pass. This, in turn, creates a ripple effect for teachers, especially if their performance is measured by pass rates. They will inevitably adapt their teaching methods to align with that target.

If you dont have time to read this months blog – listen to my AI alter ego summarise the key points.

“Measure what is measurable and make measurable what is not so” Galileo

Having spent over 30 years helping accountancy students pass high-stakes exams, I’m not personally going to criticise this (exam driven) approach, though I appreciate many will. I have seen how effective it can be in providing clarity and focus for students, helping them manage the huge amounts of information they are required to learn, and in many ways making what might seem impossible – possible.

But this isn’t an argument for ‘teaching to the test’, it’s an acknowledgment of a fundamental truth that targets shape behaviour, regardless of how well intentioned the original objectives. Keep this in mind as we discuss one of the solutions currently being proposed to improve assessment, it’s called – Authentic Assessment.

“If you can’t measure it, you can’t improve it” Peter Drucker

Authentic assessment
First a definition, authentic assessment (Grant Wiggins 1989) involves evaluating learners through realistic tasks that reflect the types of challenges they will face in the workplace. It prioritises realism, and encourages learners to “do” the subject, mirroring or simulating the real-world. These assessments measure a learner’s ability to apply knowledge and skills to complex, realistic tasks. There are two main components, one a real-life task that needs to be completed by the learner and two, a rubric by which their performance can be measured.

Beyond the exam room – Instead of asking learners to sit in a room for two maybe three hours regurgitating memorised content, an authentic assessment might need the learner to prepare a portfolio, complete a project, engage in a debate, and even enter a realistic business environment, as required for a case study or simulation. Many consider them more engaging and motivational largely because learners can appreciate their real-world application. They also foster the development of critical thinking, problem-solving and ease the transition into the workplace.

“Not everything that can be counted counts, and not everything that counts can be counted” Albert Einstein

But they are not a panacea – and there area number of problems with authentic assessments, not least the lack of consistency in measuring performance. This is because they prioritises validity (measuring what it’s supposed to) over reliability (consistent results – marking scripts to the same standard). The issue with reliabilty is partly the result of not having a “perfect” model answer that can be used as a benchmark, nor a sufficiently robust rubric (marking guide). Traditional exams by way of contrast aim for high statistical reliability through objective scoring but allow little flexibility in rewarding learners who produce creative or original answers and arguably benefit those who are just good at exams.

In addition to the difficulty in objectively measuring success, as a method of assessment they are not well defined, for example what exactly is the “real world,” and what is meant by “authentic.” One company’s definition of the workplace will be very different from another.

One final observation, although authentic assessment has been proven to increase employability type skills, there is no evidence to show that having been assessed in this way increases you chances of getting a job!

What gets measured……
Changing the assessment directly alters the focus for both learners and teachers, shifting it towards real-world tasks means that learners are less a consumer of information but active participants in its use. For teachers, instead of concentrating on delivering content their role becomes more facilitative, guiding learners through complex tasks rather than just lecturing. And overall this change in approach is hugely positive.

But as highlighted above, it comes at a cost, there is some confusion over what authentic actually means, and measuring success becomes subjective making it difficult to mark consistently. This last point is important, because it means that one person may interpret your answer as a pass and another as a fail – it then becomes the luck of the draw!

The answer….well one of them – think of assessment not as a single exam but as a framework under which several different formats and approaches can be used. The assessment framework will expose the learner over several modules or even several years to a blend of formal (reliable) exams and authentic (valid) exams. The outcome, might just give us – the best of both worlds.

Test, Learn, Test – Pretesting is hard to believe

Posted on by

What would it be like to get into a car with someone who had never driven before and ask them to take a test.

They might of course have some prior knowledge, perhaps having seen others drive but there has been no formal driving instruction. You have to admit in principle it doesn’t seem a very good idea.

Just to be clear, the test would be similar to the one they will take after the instruction and include reversing, parallel parking, emergency stop etc. 

The idea of testing before learning is hugely counterintuitive and it won’t work or be desirable in every situation, driving a car is a good example, but read the rest of this blog and you might just change your mind.  

The pretesting effect
But first, Pretesting, this involves assessing learners on information related to upcoming material before any formal instruction. There is good reason for this from a learning perspective, the pretesting identifies the baseline level of knowledge of the learner before being taught. The teacher can then work their magic, and when we test again the learner will do much better, leading to the conclusion that learning has taken place.

The pretesting effect is different, this is where the learner who takes the test before any formal instruction performs better on subsequent tests than those who didn’t take the pretest. The implication is that the very process of pretesting improves learning.

In a typical pretesting study, one group of learners would take the pretest and the other group would not (The control). All learners then study a specific topic on which they are tested (The post test). Some questions in the post test are from the pretest but some are new, meaning they have never been seen before by either group. The findings from most studies will show that the learners who did the pretest score higher than the control group, but interestingly this is on both the pretested questions and the new ones.

Let’s just pause at that point, this is pretty strange, why would the pretested group do better? All they have done is attempt some questions that on the whole they didn’t know the answer to, which might be pretty demoralising. And it can’t be because the pretested group remembered the answers because we are told they also did better on the questions they had not seen before.

Why might this work?
There have been a lot of studies in this area in order to try to better understand what is happening, for example does the type of question make a difference, Little, J.L., Bjork, what impact does it have if the learner gets the question wrong Richland LE, Kornell N, Kao LS. Spoiler, even if the learner gets the pretest question wrong, they still perform better as a result of doing it.  A word of warning, pretesting is not better than post testing, its just better than learning the material without a pretest.

Here are a few reasons:
– Activates prior knowledge – one suggestion is that pretesting connects new information to what is already known making it more meaningful and easier to remember.
– Increases difficulty (desirable difficulty) – Introducing challenge during the pretests, recalling information before any formal instruction can improve the durability and transferability of learning.
– Increases attention and identifies importance – Being asked a question on a specific area that is not understood might cue the learner to pay greater attention when they come across it later in their studies. It also makes them more aware of the type of questions that will be asked.
– Feedback helps – As with any learning process the feedback received from the pretest can help clarify what the question was asking and of course offers up the answer.
– The hypercorrection effect – This is really interesting, errors committed with high levels of confidence are more likely to be remembered as long as the feedback is persuasive.
– Improves metacognition – Learners become more aware of what they know and what they need to learn.

When you read through this list, there is a rational and maybe it starts to make sense. The difficulty with pretesting is that it sounds so illogical that you don’t even try, why should you waste your time proving that you don’t know something. And if you take the test and get a low score as you believe you will, it’s easy to lose confidence in both yourself and the process. To get the most from this, you need to believe it’s going to work and that its worth the effort and time. It is also important to appreciate that the feeling of discomfort and not knowing is perfectly normal, and that can be hard to do, but as the evidence shows if you an do this, it will ultimately be worth it.

This is worth a read if you would like to know more. Test first, learn later: The power of pretesting to enhance learning, and watch this short video, it explains the concept really well – Pre-Testing: A Better Way to Learn.

Feedback – The breakfast of champions

Posted on by

There was an interesting piece of research that came out recently, it referred to something called “Temporary mark withholding”. This as the name might suggest is providing students with written feedback but without marks. On the face of it this might seem odd and frankly unhelpful, how can you judge your performance if you don’t know how you compare against what is expected?

To answer that question, you need to ask a far more fundamental one, what’s the purpose of giving feedback in the first place?

Feedback – task or ego
We need to separate feedback from criticism which often implies that the person giving it is trying “to find fault”, although it’s possible to make it sound a little more positive by calling it constructive criticism. In simple terms criticism is more about what was wrong in the past whilst feedback directs you towards what you should do to improve in the future. But when we are thinking in terms of learning it gets a little more complicated, Dylan William talks about whether its ego involving or task involving feedback. The first of these would include offering praise such as “well done you have produced an excellent answer” but he states this is rarely effective and can actually lower achievement. However when the feedback focuses on what the student needs to do to improve, and explains how they can do it, then you get a significant impact on student achievement.

He goes on to say that “good feedback causes thinking, the first thing a student needs to do when they receive feedback is not to react emotionally, not disengage – but think”. It might be worth adding that Dylan William is talking about the impact of feedback on student learning not on how the student might feel in terms of motivation, self-confidence etc. There is clearly a place for ego type feedback it’s just not that effective when sat alongside a direct instruction because the emotional response often blocks or detracts what needs to be understood for the student to improve.

Formative and Summative assessment
There is one last piece of information that will help us make sense of the reasons why temporary mark withholding might work, the difference between formative and summative assessment.

Summative – The purpose of summative assessment is to “sum up” student learning at the end of a chunk of learning or completion of a course and compare it against a standard, normally a pass rate. This is why exams are often criticised, it’s not that testing is bad, it’s how the results are used, often polarising and narrowing opinion as to an individual’s performance, pass and you’re a hero, fail and you’re a villain. It gets worse when you then put those results into a league table and publish them, with the winners at the top and losers at the bottom for all to see and draw often incorrect conclusions.

Summative assessment is however valuable, if you score below the target, it tells you that more effort or work is needed, also that you are not performing well on a particular topic, but it provides no guidance as to what you need to do to improve.

Formative – The purpose of formative assessment is to monitor progress on an ongoing basis in order to help the teacher identify the “gap” between what the student knows and needs to know. This is where the magic happens, firstly in finding out where the gap is e.g. Where is the student currently compared to where they need to be, then figuring out the best way of getting them to that higher standard e.g. what do they need to do to improve. Formative assessment can be a test, a quiz or simply observation.

Lessons for students
And this is why holding back the marks works, what the piece of research (et al) highlighted, is that when students get their marks, they effectively prioritise the grades over the written comments. The good students ignore the comments because they don’t think they have anything to learn, and the weaker students are demotivated so also ignore them.

The key point for students is this, by all means look at the mark but resist that emotional (ego) reaction to pat yourself on the back or beat yourself up. Read all the comments with an open mind, asking two simple questions, can I see that there is a gap between my answer and the model answer and secondly do I know exactly what to do next to close it? The feedback, if it is good of course should make this as easy a process as possible.

The fact that your script might only say “see model answer” or have a cross with the correct number written next to it, is more an example of poor marking with little or no feedback. Perhaps you should return your script providing the marker/teacher with some feedback highlighting the gap between good marking and bad marking but most importantly what they should do to improve…..

And if your interested, here is the link to Dylan William explaining the importance of formative assessment.

Reference – Kuepper-Tetzel & Gardner – Jackson & Marks, 2016 – Taras, 2001, Winstone et al., 2017 – Ramaprasad, 1983

Putting the context into case study

Posted on by

Context

I am still reading Sensemaking by Christian Madsbjerg and as I always tend to do I have been trying to reduce the 216 pages down to something that is both meaningful and memorable. The rational for this is that if I can summarise the essence of what is being said into a single statement, then my level of understanding is reasonably good, and it makes it easier for me to use what I have learned in other situations.

So here goes, if I was to summarise what Sensemaking is all about, in one word it would be..….Context. In essence, in a world of complexity and abundance of information we are in danger of thinking that the “fact” we see on our computer screen, offered up by a search engine, driven by an algorithm is the truth, when in reality it’s only one version of it. Without the context from which this information came we are fooling ourselves as to its true meaning.

As a result of this discovery, I wondered into an area I  have wanted to write about before, the importance context plays in changing what something means, especially in examinations. Getting the meaning wrong could be the reason you fail the exam rather than pass it.  Even objective tests will have some form of context setting just before the actual question. But the type of exam where you are most likely to have a problem with context, is a case study.

Jokes play with context

A hamburger and a french fry walk into a bar.

The bartender says, “I’m sorry we don’t serve food here

The importance of context in case study

I have written about case studies before, “passing case study by thinking in words,” but focussed more on the process of how you think and write rather than how you interpret the information presented.  Case studies are becoming an increasingly popular way of assessing a student’s ability to apply knowledge from several different subjects (synoptic) in the context of a real-life situation.  This shift towards case studies is understandable given the need for improved employability skills. Here is a great story to illustrate how context changes the decision you would make or as often in a case study, the advise you would give.

A battleship had been at sea on its routine manoeuvres under heavy weathers for days. The captain, who was worried about the deteriorating weather conditions, stayed on the bridge to keep an eye on all activities.

One night, the lookout on the bridge suddenly shouted, “Captain! A light, bearing on the starboard bow.”

“Is it stationary or moving astern?” the captain asked.

The lookout replied that it was stationary. This meant a collision would result unless something changed. The captain immediately ordered a signal to be sent to the other ship: “We are on a collision course. I advise you to change course 20 degrees east.”

Back came a response from the other ship: “advise you change your course 20 degrees west.”

Agitated by the arrogance of the response, the captain asked his signalman to shoot out another message: “I am the captain of one of the most powerful battleships in the British navy, you change course 20 degrees east now.”

Back came the second response: “I am a second-class seaman, you had still better change course 20 degrees west.”

The captain was furious this time! He shouted to the signalman to send back a final message: Change course 20 degrees east right now or you will leave me no choice!

Back came the flashing response: “I am a lighthouse – your move.”

How to deal with context

It is easy even in the example above to think you know what is going to happen or what you would do. But when the context is revealed, your advice fundamentally changes. Case studies are created to see how well you respond in certain situations, so it’s important not to jump to conclusions.

And this is where sensemaking plays its part, use your senses, don’t just look at what is there, think in opposites, what is not there, what’s missing? Use visualisation, see yourself in that situation, look around, free up your thoughts, what do you see now? But most of all, be curious, ask questions of the scenario, how big is the ship, how long has the captain been in charge, what is the weather like, are there others close by?

Another excellent tool to use in these situations is called perceptual positions. Think of the event from different positions, firstly yours, what does the event look like through your eyes, secondly, the other person(s), what would you do if you were them, and thirdly what would the event look like if someone was looking in, observing both parties.

Case studies in the future will become even more sophisticated. Virtual reality offers up so many opportunities to create real world environments in which to tests students. And when that happens, you will definitely need to use all of your senses to get you through – take a look at this 360 VR surgical training, amazing.

And one last joke

Thomas Edison walks into a bar and orders a beer.

The bartender says, “Okay, I’ll serve you a beer, just don’t get any ideas.”

 

 

 

 

 

 

Teaching to the test – another point of view

Posted on by

point-of-veiw-2A point of view is a programme on radio 4 that allows certain well-read, highly educated individuals, usually with large vocabularies to express an opinion. It lasts 10 minutes and is often thought provoking, concluding with a rhetorical question that has no answer.

This week Will Self the novelist and journalist gave his point of view on teaching to the test, as you might imagine it caught my attention. Self starts by telling a story about the life of a “good student,” and how it would unfold. He describes the way in which their concentration intensifies when the teacher states that what they are going to learn next is important and often examined. The story continues, as a result of their diligence and technique, the “good student” gets the necessary grades to go to University. They don’t however select the University on the basis of the course of study or on what they passionately wish to learn, no its based on the Universities credibility in league tables.

Upon successfully gaining a degree the student, now an employee gets a desk job that rewards a similar style of rubric mentality. As an employee, they are assessed against targets, performing well only on the ones that promise promotion and a pay rise. Eventually they retire and die.

Self concludes that this ordinary, dull, uninspiring life started back in the classroom all those years ago, when the teacher failed to educate and inspire, and simply taught to the test.

Over egging the pudding

There is a logic to this story, and it sounds ever the more inevitable as Self narrates it in his black and grey voice. But that’s all it is, a story. It avoids detail and colour, offering little regard as to the individual’s ability to reflect at some point in their life and ask searching and probing questions. It is as if somehow because the teacher highlighted the importance of one piece of knowledge it somehow stifled the student’s capacity to one day think for themselves.  Self is how they say, overegging the pudding, taking an interesting question as to the impact teaching to the test might have and serving up an omelette.

Teaching to the test is not bad

Brunel university asked a question as to what makes an unmissable lecture. In addition to many arguably more commendable answers, including the passion of the tutor and because they wanted to learn, the likelihood of the subject being taught having a high probability of being in the exam was key. Suggesting that a specific topic might be on the exam paper firstly, ensured a good attendance and secondly guaranteed the student listened intently.

Attention is important but even for the diligent student focus is vital. Learning everything is simply not possible, faced with 20 chapters, the student needs some clue as to where they need to direct their energy and time. Of course, the educationist will say that everything is important, but saying that will not make it so. Knowing that something is examinable at least gives a starting point and helps guide the student through the material quickly and efficiently. It’s also worth adding that It does not exclude the need to be inquisitive, in fact by making the student read a particular topic it may inspire them to find out more.

Exams and exam answers also provide examples of what is expected and the standard the student must reach if they are to be successful, no amount of narrative in the student handbook or curriculum guidance will do this as effectively.

The type of assessment matters

Of course, in Selfs world, teaching to the test removes the need to do anything more than learn about what will be in the exam. He suggests that students need to think outside the box rather than simply tick them. I have to admit I like that sentence.

But he does have a point, if the test is so narrow that it only assesses memory or a very small part of the syllabus then that is all the student will focus on. But that is just a bad test, this is of course where I am in danger of becoming idealistic and painting a picture that is not a true reflection as to what is happening. Not all tests are good, and undoubtedly some students will pass with limited thought and little more than good memory skills. Yet with changes in technology it becomes ever more possible to build tests and simulations that asses the student ability to perform in real world situations, and for that matter think outside the box.

Teaching to the test has become a term used to describe bad teaching and poor assessment and no one would agree that either of these are desirable. But it is not the process that’s problematic, it’s the application. Testing in its many forms is part of learning but it needs done well and thoughtfully.

In conclusion

Having now read the blog I would encourage you to listen to Will Self – click. It is of course not for me to say who presents the right point of view, you need to make up your own mind. For those however who were taught to the test no matter how long ago, you probably won’t understand even what I am asking because to the best of my knowledge this question has never been tested before……………?

 

The Paradox of on demand exams

Posted on by

paradox-of-choice-1

But which one is best when there is so much choice?

It is more blessed to give than receive, so it says in the bible, and Christmas is a time when you get to do this in abundance. What better feeling can there be when you find exactly the right present for that special person. You can even imagine the smile on their face when on Christmas day they rip open the paper to reveal the gift they either never new they wanted but did, or in their wildest dreams believed they could have or own.paradox-wonderful-christmas-time-1

BUT…….its not as easy as it used to be. Deciding on what you want to buy is one thing, finding the “best” or most suitable gift is another.

Paradox of choice

Although Psychologist Barry Schwartz wrote his book called The Paradox of Choice over 12 years ago, most if not all of what he said is more relevant today with the explosion of choice created by the internet and advances in technology. He argued that choice rather than being good, and by the way the basis for freedom, was not necessarily bad, but did not always result in the best decisions being made or to be precise, the feeling that the best decisions had been made.

There were two reasons for this, firstly, too much choice creates paralysis, a delay in the decision-making process (see also Buridan’s ass) that can result in no decision being made at all, and secondly if you do decide, ultimately you will become less satisfied with your choice for a variety of reasons e.g. regret, opportunity cost and the escalation of expectation.

But what has this all got to do with examinations?

On demand exams result in their own paradox

In the world of professional examinations for many years’ students and employers have been asking for more choice as to when the exam can be sat. Its only tradition and inflexible systems that have created a need for testing to take place at specific times of the year. But with technology comes flexibility and that flexibility now means that some examinations can be sat when you want, not when the professional body or university decides. Driving tests have long followed the concept of, “on-demand” testing.

This choice is however resulting in a paradox. Students are taking longer to pass their exams but despite having more time results are not improving. You would expect that having additional time to study would improve exam success not reduce it.

One of the reasons for taking longer to pass is because it is now within the student’s power to change the exam date if they so wish. There are a number of good reasons for doing this, for example, work pressure, not feeling ready for the exam or that you dont know enough. Logical reasons but are they “good” reasons?

That all depends of course on the objective, but if the objective is to pass the exam quickly, then the answer is no.

The best of both worlds

Schwartz never concluded that having a choice was bad, just that it was not the end game. Ultimate choice is not the objective. He did say that we should perhaps stop thinking in terms of maximising choice but set some standards that can be used to help navigate the choices available.  In fact, this is borne out by some initial research into this area which shows that students increase their chances of passing if they set an exam date, (the choice) enable sufficient time to study but don’t change the exam date later (the standard). The choice when to pass can be set but neither the length of study nor the exam date should change.

Alternatively:

Consult an expert – What do people do when they are faced with such a wide range of options they cannot decide, how about asking an expert? In this instance the expert would be the educational establishment or the examining body, what do they recommend?

Follow the norm – Data is now more readily available than ever before, many technologies make use of historical trends to make predictions or to offer advice when there are lots of choices. Everyone will be familiar with the web sites that suggest books or other items you should buy based on past behaviours. What worked best for you in the past, what did you do before that was successful or what do the most successful people do?

Now that we have cleared up how to make better decisions in a world of endless choices – I need to begin thinking about my New Year’s resolutions, oh dear if only there were not so many choices.

Happy new Year to you all

Click to watch Barry Schwartz TED lecture.

 

 

Currywurst , apples and the £33,000 a year boarding school

Posted on by

currywurst

Currywurst fast becoming the most popular dish in Berlin

I have three relatively unrelated stories this month.

Berlin

The first follows a trip to Berlin this weekend and the obligatory city tour. Just of Unter den Linden you will find a sunken glass plate between the pavement that provides a view into a room full of empty bookshelves, sufficient to house the 20,000 books that were destroyed on this site on the 10th of May 1933. Some 70,000 people, many German university students gathered to burn books with “unGerman” ideas. Joseph Goebbels joined the students at the bonfire and declared: ” The future German man will not just be a man of books, but a man of character.” “It is to this end that we want to educate you.”

This was not only symbolic but practical. Without these books or to be precise only having the ones that remained, it might just be possible to create a nation of people who all think the same? Who all share the same ideals and live life by the same values. And yet of course this is not how you create an educated society, a society able to think for itself, the result will be compliance and idealism. Which was more likely what Goebbels had in mind.

Goodwill Hunting

When I got home one of my favourite movies had already started. Good Will Hunting tells the story of a maths genius, Will Hunting who struggles to come to terms with his past and as a result is unable to make a commitment for fear of being let down. But for me the best scene is where Will confronts a first year grad student who has been belittling his friend Chuckie.

Will: “Wood drastically underestimates the impact of social distinctions predicated upon wealth, especially inherited wealth…” You got that from Vickers. “Work in Essex County,” Page 98, right? Yeah I read that too. Were you gonna plagiarize the whole thing for us, you have any thoughts of your own on this matter? Or is that your thing, you come into a bar, you read some obscure passage and then you pretend, you pawn it off as your own, your own idea just to impress some girls? Embarrass my friend?

Will: See the sad thing about a guy like you, is in about 50 years you’re gonna start doin’ some thinkin’ on your own and you’re gonna come up with the fact that there are two certainties in life. One, don’t do that. And two, you dropped a hundred and fifty grand on a fxxxin’ education you coulda’ got for a dollar fifty in late charges at the Public Library.

Clark: Yeah, but I will have a degree, and you’ll be serving my kids fries at a drive-thru on our way to a skiing trip. Will: [smiles] Yeah, maybe. But at least I won’t be unoriginal.

This of course challenges the idea that knowledge is the same as thinking. Knowing something is not the same as having manipulated the information in your head, questioned and explored the arguments from many angles, the result should be an opinion, not someone else’s but your own.

Wellington college

And finally to bring my trilogy up to date. Only this week Wellington College who boast George Orwell and Sebastian Faulks as past students announced that have introduced a six-hour interview process to help teachers see beyond children’s over tutoring. They encourage families to spend more quality time instead of having their children tutored, Julian Thomas, Wellington’s head master, said: “When parents ask how should they prepare their children for our selection process, we tell them ‘have supper together and talk as a family; go to plays and good films and discuss them. Help them to think critically about the world around them; enjoy interesting conversations go out for walks and see the world”.

What Wellington are trying to do is encourage behaviours that will help children become thinking adults.

In conclusion

Knowledge like information is becoming increasingly accessible, unlike Germany in 1933 we have no restrictions, the internet has given us that power. But learning is more than knowledge you have to take the facts and make them your own, stitching them together carefully and thoughtfully, and that takes time and a considerable amount of effort. On the whole absorbing information will probably not come as easy to you as it did to Will Hunting, yet even with his genius he only knew what he had read. His learning had in fact only just begun as he went looking for the real life experiences that would test if he really understood anything at all.

Examinations are moving more towards tests that require you to think, asking that you provide an opinion or make recommendations given a set of circumstances, case studies are a good example. Of course you need knowledge as a foundation but the ability to sift quickly through the vast amount of information available may in time become more important than knowledge itself .

Ps apples is taken from Good Will Hunting – “How ya like those apples”

 

 

The tip of the iceberg – exam tipping is becoming obsolete

Posted on by

tip1

Assessment is changing, there was a time when all examinations were sat in a room, the answers would be hand written on a piece of paper and a retired English teacher would stand at the front reading out instructions as to what you could and couldn’t do in the next three hours.

Not any more…….you request a date that is convenient, turn up at the exam center, no longer is this a sports hall, it might be a driving school test center or the college you studied at. Then you log onto the PC and answer questions on the computer screen in front of you. The results may be immediate; it depends on if it is “human marked” or computer marked.

But in some ways these changes are only the tip of the iceberg!

What no past exam papers.

As examinations move into the digital world we are seeing other changes as well. There is a move towards objective testing, scaled scoring and examining bodies no longer providing past exam papers, what did you say, no past exam papers……!

This is partly down to the nature of the test i.e. you can’t provide an exact replica of a past exam question if it is an objective test. Remember objective test questions are randomly selected from a pool, and are different for each student. But there is also a shift towards some examining bodies only providing an example of the type of questions that could be set rather that a continuous flow of, the last exam papers.

If the test changes – how you study (and teach) has to change

Now for someone who has advocated that students analyse past exam questions in order to identify key areas so as to better direct their studies, this is a bit of a blow. It has also been the method I have used in the past to focus my own delivery in class and on line. Of course using past exam questions has always been much more than just spotting key areas, it is about focus, providing a place to start, showing content in the right context, helping with writing style etc.

There will still be past questions, sample questions will be provided. What we don’t know is how representative they will be of the examination. Or will it be as we have seen in the past with pilot and specimen papers, they change over time, drifting away from the original in terms of style and emphasis. Although I can see the logic in examining bodies not releasing papers, I hope they will continue to keep the sample papers fresh, in keeping with current thinking about the subject and how it will be examined.

What to do?

Students and tutors still need focus, there has to be emphases on key areas in order to chunk the content so that it can be more easily learned, it’s just that we won’t be able to use past questions or at least as much as we have in the past. That emphasis will now have to come from articles written by the examiners, examiner reports and syllabus weightings. If faced with a new subject where there is only one sample paper, it will be necessary to read the guidance from the body closely, noting reference to “this being a key part of the subject” or “one the examiner thought was answered badly in the past.” These together with the syllabus weightings and specific learning outcomes will have to be your guide. It is of course possible that the subject has not changed much from before and so some of the older past question can be used. As far as questions style is concerned then that will have to come from the questions and answers that are published, it may not be ideal but it’s the best we can do.

The overall impact of these changes is that students will have to know more, something that is hard to argue with. Students and tutors alike will have to devote far more time to the subject, which is fine if students have the time and can afford the extra costs involved in longer periods of study.

But it’s not all bad news, new technologies can help students make the most of dead time, studying on the train using their mobile phone for example. Also knowledge is more freely available than ever before as many top institutions provide a huge amount of free easy to access content online.

One final thought, examinations may change and they may not be fair but on the whole they are equal, everyone as before is in the same boat, and someone will always pass, wont they!

The future – Sitting the exam at home?

On line exams

An online student, all be it a mature one shows his ID to the online assessor

And maybe even the exam room will become obsolete. Proctur U is a US based company that also has a presence in the UK offering online invigilation. Watch this video to see how it works and judge for yourself

 

 

So you want to be an astronaut – assessment for astronauts

Posted on by

 

Astronaut-Tim-Peake-Dials-Wrong-Phone-Number-from-Space

Your there by yourself. There’s no doctor, there’s no computer engineer – so you have to learn all of these skills. Tim Peake

What is the purpose of assessment?

In order to gain a better understanding of the assessment process for astronauts, let’s ask a more basic question first, what is an assessment and what does it prove?

At one level it is simply a measurement of performance benchmarked against a given outcome or standard. The results can, help a teacher identify progress so they can adapt the next lesson, or give assurance an individual is capable of performing a particular task. This is probably the most important type of assessment for astronauts.   Modern high stakes “examinations however play another role; they offer the student a transferable badge of honour that can help open doors to better career prospects and increased salary. If you doubt the importance of this last point, click this link to read about the level of corruption in India and the lengths people will go in order to obtain a certificate saying they have passed an exam.

Employability gap – so what’s the problem?

However examinations on their own do not provide a guarantee that the individual who passes will be able to do the job.  And it is here where the disconnect becomes clear. Often the method of assessment is not based on what the student will be doing in the work place, and even if it is, it is not set in the same context. Many believe this gap has become ever wider as more and more students come out of school/university lacking the skills required by the employer.

But should the exam/assessment be changed to narrow the gap or is it the role of the employer to provide the necessary “on the job” education in the work place? However the problem with the employer being given the responsibility, it implies that what you study doesn’t matter, only that you do. This just seems wasteful, wasteful of time, money and effort. Assessment must get closer to measuring the skills required for one simple reason, what gets measured gets achieved, a cliché for sure but a true one.

Assessment for astronauts

Costs – The costs involved in training astronauts is a bit unclear, NASA will pay Russia $70m for a seat on Russian space craft in 2017 that includes training, figures of around $25m are also cited, Tim Peaks training was quoted as £16m so very similar.

Basic requirements – Mission Specialist (non pilots – most are pilots with a military background) include the following:

  1. Bachelor’s degree from an accredited institution in engineering, biological science, physical science, or mathematics. The degree must be followed by at least three years of related, progressively responsible, professional experience.
  2. Ability to pass a NASA space physical, which is similar to a military or civilian flight physical.
  3. Height between 58.5 and 76 inches.

Training – Tim Peak’s education started in 2008 when he was selected by ESA (European space agency) from 8,413 applicants. Tests at this stage included, intelligence, spatial awareness and concentration. Then for the 10 selected another 18 months of intense training followed, this time on topics as wide ranging as space law, rocket propulsion, spaceflight engineering and the hardest for Tim, learning to speak Russian. He was also subject to one week’s caving in Sardinia, Italy, with five other astronauts to simulate what it would be like in space, this was to build teamwork, problem solving and cope with poor hygiene facilities. Then it was one year of advanced training, including working underwater to simulate the lack of gravity. In 2013 it was time to go into space for the first time, this was the last test before being selected and then finally in 2015 Tim was allowed to do the job for real. In all it had taken 6/7 years.

What have we learned?

Clearly you need some underpinning academic skills i.e. a degree but a relevant one. The training is provided by the state, the argument being that it is for the greater good of society so the costs should be met by the tax payer. An interesting point for those that believe this should be the case for all education. But most importantly the assessments were all built around making sure that when Tim got into the space he could do the job. Very few of the tests required a piece of paper and a desk, and many were simulations of what he would meet in the real world.

Given the advances in technology the time is now right to introduce more simulations into the exam room, not so we can all become astronauts but to help prepare the next generation for the work place.

And just imagine your badge of honour when going for that next job – “what exams have you passed” “oh a few, but did I mention I was an astronaut…..”

 

The mystery of scaled scoring

Posted on by

A mystery is defined as something that is difficult to understand and or explain. This describes exactly the situation we have at the moment as many examining bodies move their assessment towards scaled scoring and away from the more traditional percentage pass mark.  Some argue that scaled scoring is clearer in that it allows direct comparison of one examination to another but from my experience in the short run at least clarity is not the term many would use.

The raw score

How did I get on?For most students the way to find out how well they are doing is to have a piece of work marked. If the total marks available are 50 and the student scores 35 then as a percentage they have obtained 70% of the available marks. This is called the raw score. Interestingly it tells you very little, for example is 70% good or bad? For a raw score to have any meaning it needs to be compared with a standard. This could be with another student or group of students, and is an example of what is called norm referencing, now this clearly has meaning as you can tell if you are performing better or worse than your peer group. But the most common way in which exam results have historically been published is as a pass/fail against a pass mark or as it is sometimes called a cut score. If the pass mark is 60% and the student has scored 70% then they have passed.

The scaled score

However raw scores are only fair if the exam sat is the same for every student. It would be unfair to give each student a different exam and set the same pass mark, unless of course you believed the exam was of an equal standard.  This has been the argument in the past, yet producing an exam of equal standard is almost impossible, and this is why scaled scores are being used more often.

To use scaled scoring you need a range of marks, for our purposes let us assume the range or scale goes from 0 to 150. We then have to set a common passing score, say 100, this is the score on the scale that the student needs to achieve to pass the exam or test, and it will never change. What happens next is what makes a scaled score so useful.

ExpertsA group of experts including the examiner review the exam that has just been sat, they will probably mark a number of scripts themselves and will have access to the results from all other students. Then taking into account the performance of all students and their knowledge of this particular exam they will set the pass mark for this paper. Let’s assume that the pass mark is 32, if a student is awarded a raw score of 32 on this particular paper it will be equated to a scaled score of 100. All other scores above and below 32 will also be given a scaled score equivalent.

So on some levels using a scaled score is no different to the examiner setting a different pass mark for each exam, acknowledging that the level of difficulty will never be the same for any one examination. The reason the scaling is used however is because it can be stated that the passing score for the exam is always 100, this means the standard stays the same even though the marks may change.

So there you have it, no longer a mystery….I hope?

For more information and guidance watch the video on Pearson’s web site. Click on this link and scroll down.

 

 

 

 

 

 

 

 

 

Playing the game – Tips for answering objective tests

Posted on by

Amazon

Which is the real logo?

Objective tests (OT’s) are becoming increasingly popular.

There are many reasons for this; educationally one of the big advantages is because, they are objective i.e. the mark is accurate, it’s either right or wrong.

It may come as a surprise to some students but when people mark there will be a slight (we hope) difference, its called marker bias.

The more cynical might argue that the examining body introduces them to simply save money. Whatever the justification students will find OT’s being used more frequently, at all levels and so need to be prepared.

But OT’s must be easier?

Answering a question like…. which one of the following best explains how a car engine works a-b-c-d, must be easier than writing/typing an answer explaining how a car engine works – right?

Well perhaps not, what if all of the answers are plausible? Think of the mental process you go through trying to distinguish between them, digging deep into your understating, looking for a word in the question that might give a clue, might help you narrow down your options. And even if you do narrow them down to let us say, 50:50, there are no method marks, it’s either right or wrong; it’s 50% of a full mark or 50% of no mark. The student answering the written question is unlikely to get no marks at all even on a question they don’t really understand.

Personally I think OT questions are more difficult from a student’s perspective but very useful for examining bodies as part of the assessment process. But they should only be part of that assessment process; other types of assessment should also be used.

Playing the OT game – Tips

But what can be done, how can you improve your chances of passing OT’s?

Well there are the simple things like, make sure you read the question carefully. This is much easier if it is a paper based test where you can underline exactly what you have been asked to do. I have written in the past about the importance of underlining. It helps the brain focus on what is important and what is not.  This is made much harder when the OT questions are on the screen. In these circumstances I would suggest you write out the key words on a pad or white board.multiple-choice marriage'

Answer the questions you can answer first and leave the longer more debatable questions until the end, and follow the advice of Ludy T. Benjamin, et. al (1984). She identified you are better changing your original answer to another one if you doubt it. This is very much the opposite of conventional wisdom that suggests the first answer you come up with is probably correct. But be careful, this is only if you doubt your original answer. The argument being that when looking at the question a second time you can tell something isn’t right and so will spend more time on the question than before, changing to a more plausible one.

There are some more sophisticated techniques that can help reduce the odds, I have summarised them below. Many of these I have borrowed from a more comprehensive article written by a colleague, John Bennett – thanks John.

  • Distracters – these are questions that contain an answer very similar to the real one and are often plausible. The technique is to cover up all the answers, so that they don’t distract, work out what you think the answer is, then reveal. Hopefully your answer will be in the list.
  • Go for the long answers – William Poundstone author of “Rock Breaks Scissors” noticed that the longest answer on multiple choice tests was usually correct. “Examiners have to make sure that right answers are indisputably right,” he says. “Often this demands some qualifying language. They may not try so hard with wrong answers.”
  • Eliminate the outliers – Another Poundstone tip is to look out for one of the answers that is very different to the others, and if you find the outlier it’s probably wrong. So for example if you had 4 numerical answers a£0.46 – b£0.54 – c£0.55 – d£1.60. The outlier in this sequence of numbers is d and is unlikely to be correct. It doesn’t of course give you the answer but it will at least improve your odds.
  • Find opposites – An easy one next, where two answers are exact opposites, the answer is more likely to be one of those two.
  • Look out for general words – The University of Minnesota identified that the question that use general words such as, mostly, possibly, often, usually, will “often” be the correct one.  This is because when an examiner wants to write an incorrect answer they will be far more specific e.g. it will NEVER rain on Friday as opposed to Friday is OFTEN the wettest day of the week.
  • Negative worded questions – when questions ask which of the following is NOT true mark off the ones that ARE true first. The brain struggles to recognise negatives, so you need to put the question in terms of positives, what it is as appose to what it is not.

There are of course more techniques but you don’t want to enter the exam worrying more about the techniques than the exam itself.

And don’t forget hard work – exam tips only stop you failing

And as you would expect me to say, these tips only help you play the game better, hard work, studying and practising questions are far more important.

The answer is 3 – Amazon sells everything from A – Z and that puts a smile on your face

Are exams fit for purpose (part two) – what are the alternatives?

Posted on by

You dont fatten pigs 2

Last month’s blog came to the conclusion that examinations* are fit for purpose or at least “a purpose.”

They provide the student with a clear objective to which they can direct their efforts and focus attention and are a transferable measure of competency that can be assessed at scale. The “at scale” point is important as there are many ways of assessing competence but few that can cope with the need to test thousands of students all at the same time.

The main problem with examinations is that they don’t always examine what is most valued; the method of assessment often has significant limitations as to what it actually tests and the results are presented in league tables that give a far too simplistic view of success.

I am not sure we can resolve all of these but it might be worth exploring other options, specifically alternative methods of assessment. For example If you change the method of assessment from a formal, often timed written exam to say a portfolio of work, not only do you change the method of assessment but you will change what is being examined, two birds with one stone perhaps.

Different methods of assessing competence

Open book exams

Open book assessment offers a way of testing application rather than memory. Students have access to a text book that contains information relevant to what they are being asked. It’s the use of knowledge that is important, not the knowledge itself. The idea of open book could easily be adapted, why not allow students access to the internet during the exam, they could look up anything they wanted. Is this not more representative of what happens in the real world?

Take out exams

Similar to the above the so called “take out exam” allows the student to take the exam away to work on at home using whatever resources they prefer, books, internet etc. They return the next day with a completed answer. This can work better than you might at first think so long as you have a robust mechanism to detect plagiarism. There are several very good software packages that can spot the most sophisticated types of copying.

Case studies/simulations

A case study provides an environment for the student to demonstrate they can use their knowledge to solve problems and or offer advice in a virtual world. Most case studies tend to be written but this is one area that we could see some clever and affordable use of technology to better simulate the real world.

Performance tests

In a performance test students are required to demonstrate a skill/process, create a product etc while being observed by the assessor who will evaluate the performance. A great example of testing ability to apply knowledge but suffers from the subjectivity of the assessor and has limited application at scale.

Portfolios

Portfolios are most often collections of the student’s work that demonstrate their ability to perform a specific task. These can be simulations of the real world or portfolios of work actually undertaken on the job. A portfolio can include written documents, emails, audio or video recordings, in fact anything that provides evidence as required by the assessor.   Portfolios are perfect for assessing application but the process of assessment is expensive and not without bias.

Viva Vocal – (living voice) Oral exam

Often used to test PhD students, an oral exam gives the assessor chance to question the student. This is a very effective method where you are looking for higher level skill and depth of understanding. As identified last month it’s probably one of the oldest forms of assessment.

Digital badges – capturing the learning path

Being awarded a badge as recognition of achievement is something many will be familiar with especially if you were a boy scout or girl guide. But digital badging is new and becoming increasingly popular because of the internet. A good example would be linkedin and the badges awarded to you by others as recognition of certain skills.  Many of the assessment methods above provide a first past the post type of assessment, you pass and that’s it. Digital badging on the other hand is a form of lifelong assessment that evolves along with your career.

Digital badging for me is one of the most exiting forms of assessment and I am not alone Nasa have been using digital badging since 2011. Read more about digital badging.

Assessment in the future

Scanning for competenceThe list above is far from comprehensive and many other equally valid types of assessment exist e.g.  Role plays, Slide presentations, Assignments etc but what might assessment look like 15 years from now. Well how about using MRI scans to identify which parts of the brain are being used?  Not sure it will catch on but it would provide some interesting evidence as to how the student is getting to the answer, simple memory or a genuine and deep understanding .

*Examinations defined as a written test administered to assess someone’s level of understanding, knowledge or skills

50 Shades Darker – How much to test

Posted on by

How much to test

In August last year I wrote how Professional accountancy bodies believe that competency can be measured by a candidate scoring 50% and failure if scoring 49%. This all or nothing approach seems neither fair nor useful; hence the idea that grades of competence could be introduced e.g. 50% pass, 65% commendation, 75% distinction etc.

But the mark a student gets in an exam is only part of the story when it comes to measuring competence. Can a student be considered competent if the exam they pass only includes questions from say 75% of the syllabus? Yes the whole syllabus might be covered in an 18 month period but in any one exam 25% is not tested. Equally the 75% is often considered core and so examined every sitting, this means that a student need only focus on the 75%. Admittedly if the pass mark is 50% they need to score 50% out of 75% (67%) but with practice this is possible. One final observation, it is unlikely the student will score 0% on the non-core part of the syllabus. They may get say 5% or even 10% out of 25%. Not a great score but the 67% pass mark now becomes 58%. This logic sits at the heart of the exam driven approach.

Objective testing might be the answer?

Objective tests (OT) – test that are relatively short and can be unambiguously marked, are considered by some to be a weaker form of assessment, they are part of the dumbing down of examinations. The beauty of an OT question is that the marking is completely accurate, no marking bias at all. This is often ignored in traditional exams and not seen as a problem largely because it’s not that visible. But the OT does not solve the “how much to test” problem, in fact it makes it worse. If you are asking for less then you are examining less. So if the only benefit is the avoidance of marker bias why are more examining bodies using OT style exams, is it just about saving money…….?

An example

Imagine that you have 4,500 OT style questions that cover every aspect of the syllabus, let’s also assume that the student only has to answer 50. The 50 questions are randomly picked from the 4,500 in the question bank. Is it fair that a student is considered competent if they are only being tested on 1% of what they need to know?

I think the answer is Yes, because in order to be sufficiently prepared to answer 50 questions from a bank of 4,500 when there are no core topics you have to have be capable of getting all 50 correct (assuming a 100% pass mark) and because you don’t know which 50 are coming up this effectively means you have to be able to get all 4,500 correct. The examining body can of course control  the effectiveness and level of difficulty by changing the pass mark.

All OTs would be OTT

This is not an argument to suggest that all examinations should be assessed using OT type questions, they should not. For example they are not Less-is-Morevery effective at measuring a student’s ability to communicate or evaluate complex and ambiguous situations, but they should be considered part of the tool kit that examining bodies have in assessing competence.

So on the face of it OT’s may look like a soft option, anyone can tick a box but they are certainly not easy to get right. Maybe less is actually more…..

Thinking in box’s – Cracking case study

Posted on by

Put it in a boxExaminations come in many shapes and sizes, short form, multiple choice, essay, case study etc. I know there are other methods of assessment but I am thinking here of the most common. Of these one stands out as being very different, the case study. Developed by Harvard in the 1920s the  case study involves giving the student a  real life, normally business situation and asking relatively broad Socratic type questions e.g. what do you think, why etc.

It not only puts the student in a realistic situation but also requires them to think far more deeply.  The cognitive process involved in answering a question such as what is the capital of France or can you add 2+2, on the whole is very simple and may need little more than memory. However giving student a real life business to analyse and asking them to give an opinion as to what the company should do next requires higher level thinking as well as effective communication skills.

Too much to read

One other aspect of a case study is that it often involves large amounts of narrative, all will need reading digesting and putting into context. On the face of it this can seem daunting, but it can be done and as with so many aspects of learning there is a process that can help. I have written about, having too much to read and the benefits of chunking before but I want to bring these together with another powerful technique “thinking in box’s”.

Thinking in box’s 

Volume and lack of direction is the main problem here, so we need to find our own direction and reduce the volume. Thinking in box’s refers to the natural process we have in compartmentalising thoughts. In order to make sense of the world we often put “stuff” into box’s, work, study, relationships etc. We can then open the box’s when we are best able to deal with them. The  point being that  we can’t deal with everything all at once. If this whole idea sounds a bit odd, then just consider the saying “Thinking outside the box”. This refers to the imaginary frame we put around something that restricts our ability to solve a problem and think more creatively. Strange isn’t it…..

Case study

Imagine that you have 10 pages of narrative to read based on a particular industry, a case study. There are a few sub headings and some paragraphing. You are required to provide guidance to the board of Directors as to what the companies strategy should be in the next five years.

The process 

In order to give the advice required by this question, you need to fully grasp the current situation, which means you have to  read, understand and comprehend what is written on the 10 pages. To add structure to the case firstly take chunks of content and put a frame around it, this will help focus just on this chunk of information, it also reduces the volume. A chunk will often be information under a heading or specific paragraphs. Once you have the content in a box, sift through it looking for the “key words” and underline them.  Focusing only on the key words but taking into account the context, ask yourself, what do I think about this? What does it mean, what is it telling me etc. Then write down your thoughts. Do this for every chunk of information, then number each chunk.

At the end of this process you should have read and thought about each chunk, captured those thoughts and have a numerical reference by which to structure them. The final part of the process is to read each of those chunks again and produce a SWOT. This brings all 10 pages down to just one. And by using the SWOT supported by your detailed analysis you should be able to give the advice required by the question.

watch_this_videoIn this short video I demonstrate the thinking in box’s approach.

      Thinking in box’s

And finally – A few words from Terry Pratchet

I will be more enthusiastic about thinking outside the box when there is some evidence of thinking inside it!

50 shades of Grade…..Measuring students worth

Posted on by

I don’t often write specifically about the market that I am most closely involved, that of accountancy training and education. But there does seem to be an anomaly in the way the accountancy world measure success that isn’t the case in many other professions and examinations.

To become a qualified accountant in the UK (ACCA, ICAEW,CIMA etc,  yes there are more) you have to pass a number of demanding examinations and submit evidence of practical experience. The exams are taken over 3 years and everything rests on a number of exams of 100 marks each, of which you need to get 50%. The pass rate for accountancy students varies from paper to paper but is around 60% In training and education terms these exams are often referred to as High stake exams. For some failure is not simply a setback in terms of time, it could result in a lost job.

50% good – 49% bad

The purpose of such exams (including the practical experience) is of course to ensure that those that pass are competent. But how competent….?  It seems that you are either competent or not, if you get 49% you are not competent, if you get 50% then you are. This makes the margin for error very high, many a career has taken a dramatic turn based on 1 mark. Now of course in any exam there has to be a point where someone succeeds and another fails but does competence not have many shades to it, is it not a continuous process rather than a discreet one.

Shades of Grade

You may be one of thousands of students (>300,000 actually) who have just had your A level results. But of course you don’t just pass or fail, in fact this has not been the case since 1963, even before then there was an indication of the mark. You are given a grade from A through to E and all are considered a pass, 98.1% passed their A levels this year.

And the grades refer to the marks e.g. E is 40%, a D is 50%, a C is 60%, a B is 70% and an A is 80%. A similar grading exists for Degrees, 2.1, 2.2 etc and for Law exams, Pass, Commendation, Distinction.

No pressure for excellence

So why is this not the case in accountancy? To be honest I am not sure. But by introducing grades above a pass you not only provide employers with a better indication of competency, you encourage colleges to teach to a higher and deeper level, motivated by the students who want to learn more. And that for me is the most important point; you create an incentive for students to try harder. At the moment if you are scoring 55% then why put in more work, a pass is a pass, but a distinction well that will make you stand out from the crowd.

Of course the market will adjust its perception of what a pass means, for example employers may only want students with a distinction rather than a simple pass, but they have always looked and will continue to look for differentiation, at least this is a meaningful one.

What it does not do is fix the 49% you fail problem, and careers will still hinge on 1 mark, but that unfortunately is the way with exams and the reason exam technique is so important.

So to all the accountancy examining bodies why not introduce grades – grey is good and isn’t it an accountants favourite colour?

Related articles