Putting the context into case study

Context

I am still reading Sensemaking by Christian Madsbjerg and as I always tend to do I have been trying to reduce the 216 pages down to something that is both meaningful and memorable. The rational for this is that if I can summarise the essence of what is being said into a single statement, then my level of understanding is reasonably good, and it makes it easier for me to use what I have learned in other situations.

So here goes, if I was to summarise what Sensemaking is all about, in one word it would be..….Context. In essence, in a world of complexity and abundance of information we are in danger of thinking that the “fact” we see on our computer screen, offered up by a search engine, driven by an algorithm is the truth, when in reality it’s only one version of it. Without the context from which this information came we are fooling ourselves as to its true meaning.

As a result of this discovery, I wondered into an area I  have wanted to write about before, the importance context plays in changing what something means, especially in examinations. Getting the meaning wrong could be the reason you fail the exam rather than pass it.  Even objective tests will have some form of context setting just before the actual question. But the type of exam where you are most likely to have a problem with context, is a case study.

Jokes play with context

A hamburger and a french fry walk into a bar.

The bartender says, “I’m sorry we don’t serve food here

The importance of context in case study

I have written about case studies before, “passing case study by thinking in words,” but focussed more on the process of how you think and write rather than how you interpret the information presented.  Case studies are becoming an increasingly popular way of assessing a student’s ability to apply knowledge from several different subjects (synoptic) in the context of a real-life situation.  This shift towards case studies is understandable given the need for improved employability skills. Here is a great story to illustrate how context changes the decision you would make or as often in a case study, the advise you would give.

A battleship had been at sea on its routine manoeuvres under heavy weathers for days. The captain, who was worried about the deteriorating weather conditions, stayed on the bridge to keep an eye on all activities.

One night, the lookout on the bridge suddenly shouted, “Captain! A light, bearing on the starboard bow.”

“Is it stationary or moving astern?” the captain asked.

The lookout replied that it was stationary. This meant a collision would result unless something changed. The captain immediately ordered a signal to be sent to the other ship: “We are on a collision course. I advise you to change course 20 degrees east.”

Back came a response from the other ship: “advise you change your course 20 degrees west.”

Agitated by the arrogance of the response, the captain asked his signalman to shoot out another message: “I am the captain of one of the most powerful battleships in the British navy, you change course 20 degrees east now.”

Back came the second response: “I am a second-class seaman, you had still better change course 20 degrees west.”

The captain was furious this time! He shouted to the signalman to send back a final message: Change course 20 degrees east right now or you will leave me no choice!

Back came the flashing response: “I am a lighthouse – your move.”

How to deal with context

It is easy even in the example above to think you know what is going to happen or what you would do. But when the context is revealed, your advice fundamentally changes. Case studies are created to see how well you respond in certain situations, so it’s important not to jump to conclusions.

And this is where sensemaking plays its part, use your senses, don’t just look at what is there, think in opposites, what is not there, what’s missing? Use visualisation, see yourself in that situation, look around, free up your thoughts, what do you see now? But most of all, be curious, ask questions of the scenario, how big is the ship, how long has the captain been in charge, what is the weather like, are there others close by?

Another excellent tool to use in these situations is called perceptual positions. Think of the event from different positions, firstly yours, what does the event look like through your eyes, secondly, the other person(s), what would you do if you were them, and thirdly what would the event look like if someone was looking in, observing both parties.

Case studies in the future will become even more sophisticated. Virtual reality offers up so many opportunities to create real world environments in which to tests students. And when that happens, you will definitely need to use all of your senses to get you through – take a look at this 360 VR surgical training, amazing.

And one last joke

Thomas Edison walks into a bar and orders a beer.

The bartender says, “Okay, I’ll serve you a beer, just don’t get any ideas.”

 

 

 

 

 

 

Currywurst , apples and the £33,000 a year boarding school

currywurst

Currywurst fast becoming the most popular dish in Berlin

I have three relatively unrelated stories this month.

Berlin

The first follows a trip to Berlin this weekend and the obligatory city tour. Just of Unter den Linden you will find a sunken glass plate between the pavement that provides a view into a room full of empty bookshelves, sufficient to house the 20,000 books that were destroyed on this site on the 10th of May 1933. Some 70,000 people, many German university students gathered to burn books with “unGerman” ideas. Joseph Goebbels joined the students at the bonfire and declared: ” The future German man will not just be a man of books, but a man of character.” “It is to this end that we want to educate you.”

This was not only symbolic but practical. Without these books or to be precise only having the ones that remained, it might just be possible to create a nation of people who all think the same? Who all share the same ideals and live life by the same values. And yet of course this is not how you create an educated society, a society able to think for itself, the result will be compliance and idealism. Which was more likely what Goebbels had in mind.

Goodwill Hunting

When I got home one of my favourite movies had already started. Good Will Hunting tells the story of a maths genius, Will Hunting who struggles to come to terms with his past and as a result is unable to make a commitment for fear of being let down. But for me the best scene is where Will confronts a first year grad student who has been belittling his friend Chuckie.

Will: “Wood drastically underestimates the impact of social distinctions predicated upon wealth, especially inherited wealth…” You got that from Vickers. “Work in Essex County,” Page 98, right? Yeah I read that too. Were you gonna plagiarize the whole thing for us, you have any thoughts of your own on this matter? Or is that your thing, you come into a bar, you read some obscure passage and then you pretend, you pawn it off as your own, your own idea just to impress some girls? Embarrass my friend?

Will: See the sad thing about a guy like you, is in about 50 years you’re gonna start doin’ some thinkin’ on your own and you’re gonna come up with the fact that there are two certainties in life. One, don’t do that. And two, you dropped a hundred and fifty grand on a fxxxin’ education you coulda’ got for a dollar fifty in late charges at the Public Library.

Clark: Yeah, but I will have a degree, and you’ll be serving my kids fries at a drive-thru on our way to a skiing trip. Will: [smiles] Yeah, maybe. But at least I won’t be unoriginal.

This of course challenges the idea that knowledge is the same as thinking. Knowing something is not the same as having manipulated the information in your head, questioned and explored the arguments from many angles, the result should be an opinion, not someone else’s but your own.

Wellington college

And finally to bring my trilogy up to date. Only this week Wellington College who boast George Orwell and Sebastian Faulks as past students announced that have introduced a six-hour interview process to help teachers see beyond children’s over tutoring. They encourage families to spend more quality time instead of having their children tutored, Julian Thomas, Wellington’s head master, said: “When parents ask how should they prepare their children for our selection process, we tell them ‘have supper together and talk as a family; go to plays and good films and discuss them. Help them to think critically about the world around them; enjoy interesting conversations go out for walks and see the world”.

What Wellington are trying to do is encourage behaviours that will help children become thinking adults.

In conclusion

Knowledge like information is becoming increasingly accessible, unlike Germany in 1933 we have no restrictions, the internet has given us that power. But learning is more than knowledge you have to take the facts and make them your own, stitching them together carefully and thoughtfully, and that takes time and a considerable amount of effort. On the whole absorbing information will probably not come as easy to you as it did to Will Hunting, yet even with his genius he only knew what he had read. His learning had in fact only just begun as he went looking for the real life experiences that would test if he really understood anything at all.

Examinations are moving more towards tests that require you to think, asking that you provide an opinion or make recommendations given a set of circumstances, case studies are a good example. Of course you need knowledge as a foundation but the ability to sift quickly through the vast amount of information available may in time become more important than knowledge itself .

Ps apples is taken from Good Will Hunting – “How ya like those apples”

 

 

The tip of the iceberg – exam tipping is becoming obsolete

tip1

Assessment is changing, there was a time when all examinations were sat in a room, the answers would be hand written on a piece of paper and a retired English teacher would stand at the front reading out instructions as to what you could and couldn’t do in the next three hours.

Not any more…….you request a date that is convenient, turn up at the exam center, no longer is this a sports hall, it might be a driving school test center or the college you studied at. Then you log onto the PC and answer questions on the computer screen in front of you. The results may be immediate; it depends on if it is “human marked” or computer marked.

But in some ways these changes are only the tip of the iceberg!

What no past exam papers.

As examinations move into the digital world we are seeing other changes as well. There is a move towards objective testing, scaled scoring and examining bodies no longer providing past exam papers, what did you say, no past exam papers……!

This is partly down to the nature of the test i.e. you can’t provide an exact replica of a past exam question if it is an objective test. Remember objective test questions are randomly selected from a pool, and are different for each student. But there is also a shift towards some examining bodies only providing an example of the type of questions that could be set rather that a continuous flow of, the last exam papers.

If the test changes – how you study (and teach) has to change

Now for someone who has advocated that students analyse past exam questions in order to identify key areas so as to better direct their studies, this is a bit of a blow. It has also been the method I have used in the past to focus my own delivery in class and on line. Of course using past exam questions has always been much more than just spotting key areas, it is about focus, providing a place to start, showing content in the right context, helping with writing style etc.

There will still be past questions, sample questions will be provided. What we don’t know is how representative they will be of the examination. Or will it be as we have seen in the past with pilot and specimen papers, they change over time, drifting away from the original in terms of style and emphasis. Although I can see the logic in examining bodies not releasing papers, I hope they will continue to keep the sample papers fresh, in keeping with current thinking about the subject and how it will be examined.

What to do?

Students and tutors still need focus, there has to be emphases on key areas in order to chunk the content so that it can be more easily learned, it’s just that we won’t be able to use past questions or at least as much as we have in the past. That emphasis will now have to come from articles written by the examiners, examiner reports and syllabus weightings. If faced with a new subject where there is only one sample paper, it will be necessary to read the guidance from the body closely, noting reference to “this being a key part of the subject” or “one the examiner thought was answered badly in the past.” These together with the syllabus weightings and specific learning outcomes will have to be your guide. It is of course possible that the subject has not changed much from before and so some of the older past question can be used. As far as questions style is concerned then that will have to come from the questions and answers that are published, it may not be ideal but it’s the best we can do.

The overall impact of these changes is that students will have to know more, something that is hard to argue with. Students and tutors alike will have to devote far more time to the subject, which is fine if students have the time and can afford the extra costs involved in longer periods of study.

But it’s not all bad news, new technologies can help students make the most of dead time, studying on the train using their mobile phone for example. Also knowledge is more freely available than ever before as many top institutions provide a huge amount of free easy to access content online.

One final thought, examinations may change and they may not be fair but on the whole they are equal, everyone as before is in the same boat, and someone will always pass, wont they!

The future – Sitting the exam at home?

On line exams

An online student, all be it a mature one shows his ID to the online assessor

And maybe even the exam room will become obsolete. Proctur U is a US based company that also has a presence in the UK offering online invigilation. Watch this video to see how it works and judge for yourself

 

 

The mystery of scaled scoring

A mystery is defined as something that is difficult to understand and or explain. This describes exactly the situation we have at the moment as many examining bodies move their assessment towards scaled scoring and away from the more traditional percentage pass mark.  Some argue that scaled scoring is clearer in that it allows direct comparison of one examination to another but from my experience in the short run at least clarity is not the term many would use.

The raw score

How did I get on?For most students the way to find out how well they are doing is to have a piece of work marked. If the total marks available are 50 and the student scores 35 then as a percentage they have obtained 70% of the available marks. This is called the raw score. Interestingly it tells you very little, for example is 70% good or bad? For a raw score to have any meaning it needs to be compared with a standard. This could be with another student or group of students, and is an example of what is called norm referencing, now this clearly has meaning as you can tell if you are performing better or worse than your peer group. But the most common way in which exam results have historically been published is as a pass/fail against a pass mark or as it is sometimes called a cut score. If the pass mark is 60% and the student has scored 70% then they have passed.

The scaled score

However raw scores are only fair if the exam sat is the same for every student. It would be unfair to give each student a different exam and set the same pass mark, unless of course you believed the exam was of an equal standard.  This has been the argument in the past, yet producing an exam of equal standard is almost impossible, and this is why scaled scores are being used more often.

To use scaled scoring you need a range of marks, for our purposes let us assume the range or scale goes from 0 to 150. We then have to set a common passing score, say 100, this is the score on the scale that the student needs to achieve to pass the exam or test, and it will never change. What happens next is what makes a scaled score so useful.

ExpertsA group of experts including the examiner review the exam that has just been sat, they will probably mark a number of scripts themselves and will have access to the results from all other students. Then taking into account the performance of all students and their knowledge of this particular exam they will set the pass mark for this paper. Let’s assume that the pass mark is 32, if a student is awarded a raw score of 32 on this particular paper it will be equated to a scaled score of 100. All other scores above and below 32 will also be given a scaled score equivalent.

So on some levels using a scaled score is no different to the examiner setting a different pass mark for each exam, acknowledging that the level of difficulty will never be the same for any one examination. The reason the scaling is used however is because it can be stated that the passing score for the exam is always 100, this means the standard stays the same even though the marks may change.

So there you have it, no longer a mystery….I hope?

For more information and guidance watch the video on Pearson’s web site. Click on this link and scroll down.

 

 

 

 

 

 

 

 

 

Are exams fit for purpose (part two) – what are the alternatives?

You dont fatten pigs 2

Last month’s blog came to the conclusion that examinations* are fit for purpose or at least “a purpose.”

They provide the student with a clear objective to which they can direct their efforts and focus attention and are a transferable measure of competency that can be assessed at scale. The “at scale” point is important as there are many ways of assessing competence but few that can cope with the need to test thousands of students all at the same time.

The main problem with examinations is that they don’t always examine what is most valued; the method of assessment often has significant limitations as to what it actually tests and the results are presented in league tables that give a far too simplistic view of success.

I am not sure we can resolve all of these but it might be worth exploring other options, specifically alternative methods of assessment. For example If you change the method of assessment from a formal, often timed written exam to say a portfolio of work, not only do you change the method of assessment but you will change what is being examined, two birds with one stone perhaps.

Different methods of assessing competence

Open book exams

Open book assessment offers a way of testing application rather than memory. Students have access to a text book that contains information relevant to what they are being asked. It’s the use of knowledge that is important, not the knowledge itself. The idea of open book could easily be adapted, why not allow students access to the internet during the exam, they could look up anything they wanted. Is this not more representative of what happens in the real world?

Take out exams

Similar to the above the so called “take out exam” allows the student to take the exam away to work on at home using whatever resources they prefer, books, internet etc. They return the next day with a completed answer. This can work better than you might at first think so long as you have a robust mechanism to detect plagiarism. There are several very good software packages that can spot the most sophisticated types of copying.

Case studies/simulations

A case study provides an environment for the student to demonstrate they can use their knowledge to solve problems and or offer advice in a virtual world. Most case studies tend to be written but this is one area that we could see some clever and affordable use of technology to better simulate the real world.

Performance tests

In a performance test students are required to demonstrate a skill/process, create a product etc while being observed by the assessor who will evaluate the performance. A great example of testing ability to apply knowledge but suffers from the subjectivity of the assessor and has limited application at scale.

Portfolios

Portfolios are most often collections of the student’s work that demonstrate their ability to perform a specific task. These can be simulations of the real world or portfolios of work actually undertaken on the job. A portfolio can include written documents, emails, audio or video recordings, in fact anything that provides evidence as required by the assessor.   Portfolios are perfect for assessing application but the process of assessment is expensive and not without bias.

Viva Vocal – (living voice) Oral exam

Often used to test PhD students, an oral exam gives the assessor chance to question the student. This is a very effective method where you are looking for higher level skill and depth of understanding. As identified last month it’s probably one of the oldest forms of assessment.

Digital badges – capturing the learning path

Being awarded a badge as recognition of achievement is something many will be familiar with especially if you were a boy scout or girl guide. But digital badging is new and becoming increasingly popular because of the internet. A good example would be linkedin and the badges awarded to you by others as recognition of certain skills.  Many of the assessment methods above provide a first past the post type of assessment, you pass and that’s it. Digital badging on the other hand is a form of lifelong assessment that evolves along with your career.

Digital badging for me is one of the most exiting forms of assessment and I am not alone Nasa have been using digital badging since 2011. Read more about digital badging.

Assessment in the future

Scanning for competenceThe list above is far from comprehensive and many other equally valid types of assessment exist e.g.  Role plays, Slide presentations, Assignments etc but what might assessment look like 15 years from now. Well how about using MRI scans to identify which parts of the brain are being used?  Not sure it will catch on but it would provide some interesting evidence as to how the student is getting to the answer, simple memory or a genuine and deep understanding .

*Examinations defined as a written test administered to assess someone’s level of understanding, knowledge or skills

Are exams fit for purpose? (part one)

take-the-same-testI have written in the past about what passing an exam proves but have never questioned if exams achieve what they were originally designed to do, are they fit for purpose?

Firstly let me define what I mean by an exam. A written test administered to assess someone’s level of understanding, knowledge or skill that results in a qualification if successful. This is in contrast to a test which is a method of assessing someone’s level of understanding, knowledge or skill often as part of a course in order to provide feedback. A test does not have to be written. Although exams don’t have to be written either, many are and initially at least I would like to keep the definition as narrow as possible.

In order to answer the question, are exams fit for purpose we must first take a step back and look at how we got to where we are now.

 

A brief history of examinations

The first standardised test is believed to have been introduced by the Chinese in 606 AD to help select candidates for specific governmental positions. However most examinations around this time would have been oral, requiring the candidate to recite a dissertation or answer questions. Although there is evidence of written exams being used as early as 1560*, it was not until the 1820’s that many Universities began to adopt the practice. From 1850 onwards the written exam became the norm in most UK Universities. In 1854 under the Gladstone government selection of Civil servants was based on their ability to pass an exam, this time however it was written.

Bureaucracy – In 1917 to help bring some order to what had been described as chaotic the Certificate and the Higher School Certificate were introduced. Then in 1951 we had the General Certificate of Education (GCE) examinations, more commonly known as Ordinary ‘O’ level and Advanced ‘A’ level , these were normally taken at 16 and 18.

In the 1960’s the CSE (Certificate of Secondary Education) was born, opening up qualifications for all, not just those that went to Grammar school. However this two tier system was thought divisive and so in 1988 under the guidance of the then Education Secretary Sir Keith Joseph both sets of examinations were replaced by the GCSE. The GCSE was graded and contained credit for course work.  In 1991 the General National Vocational Qualifications, (GNVQS) were established intended to incorporate both academic and vocational elements, by 1995 these were accepted as ‘equivalent’ to GCSE.

In 2014 we find change again, gone is the course work and written examinations once again become the main method of assessment, although there will be grading, 1 to 9 with 9 being the higher mark. The exams will still be called GCSE’s, although officially they are known as GCSE (England). This is to avoid confusion with Wales and Northern Ireland, who are not changing.

Yes they are

Historically at least it would appear the purpose of the exam was to provide a recognised and transferable measure of competency in a given subject or discipline. The lack of transparency and consistency of the oral exam resulted in them being replaced with written ones and a more formal bureaucratic structure was developed to administer the process.

And in many ways there is very little wrong with this.

The problem is not with the exam itself, but with what is being examined. If as a society we value “thinking and creativity” for example, then should we not be examining these rather than subjects that require the candidate to do little more than rote learn facts.  Perhaps we should explore different methods of assessment, the written exam has its uses but hand written papers are looking increasingly outdated in a world that communicates electronically not only in short texts and tweets but with video and photos . In addition the way exam results are used in league tables to show winners and losers is divisive. It looks like a measure but has in fact become a target that schools and teachers must hit or be considered failures.

Please watch this it’s very funny…..and thought provoking

Not on the test

 

 

 

 

 

In the second blog about exams I want to look more closely at some of these points, in particular what other ways we can assess what people know.

*Assessment around this time was through debate between a number of learned people all at the same time and lasting for two hours or more.

50 Shades Darker – How much to test

How much to test

In August last year I wrote how Professional accountancy bodies believe that competency can be measured by a candidate scoring 50% and failure if scoring 49%. This all or nothing approach seems neither fair nor useful; hence the idea that grades of competence could be introduced e.g. 50% pass, 65% commendation, 75% distinction etc.

But the mark a student gets in an exam is only part of the story when it comes to measuring competence. Can a student be considered competent if the exam they pass only includes questions from say 75% of the syllabus? Yes the whole syllabus might be covered in an 18 month period but in any one exam 25% is not tested. Equally the 75% is often considered core and so examined every sitting, this means that a student need only focus on the 75%. Admittedly if the pass mark is 50% they need to score 50% out of 75% (67%) but with practice this is possible. One final observation, it is unlikely the student will score 0% on the non-core part of the syllabus. They may get say 5% or even 10% out of 25%. Not a great score but the 67% pass mark now becomes 58%. This logic sits at the heart of the exam driven approach.

Objective testing might be the answer?

Objective tests (OT) – test that are relatively short and can be unambiguously marked, are considered by some to be a weaker form of assessment, they are part of the dumbing down of examinations. The beauty of an OT question is that the marking is completely accurate, no marking bias at all. This is often ignored in traditional exams and not seen as a problem largely because it’s not that visible. But the OT does not solve the “how much to test” problem, in fact it makes it worse. If you are asking for less then you are examining less. So if the only benefit is the avoidance of marker bias why are more examining bodies using OT style exams, is it just about saving money…….?

An example

Imagine that you have 4,500 OT style questions that cover every aspect of the syllabus, let’s also assume that the student only has to answer 50. The 50 questions are randomly picked from the 4,500 in the question bank. Is it fair that a student is considered competent if they are only being tested on 1% of what they need to know?

I think the answer is Yes, because in order to be sufficiently prepared to answer 50 questions from a bank of 4,500 when there are no core topics you have to have be capable of getting all 50 correct (assuming a 100% pass mark) and because you don’t know which 50 are coming up this effectively means you have to be able to get all 4,500 correct. The examining body can of course control  the effectiveness and level of difficulty by changing the pass mark.

All OTs would be OTT

This is not an argument to suggest that all examinations should be assessed using OT type questions, they should not. For example they are not Less-is-Morevery effective at measuring a student’s ability to communicate or evaluate complex and ambiguous situations, but they should be considered part of the tool kit that examining bodies have in assessing competence.

So on the face of it OT’s may look like a soft option, anyone can tick a box but they are certainly not easy to get right. Maybe less is actually more…..