Category Archives: Examinations

Fairness and mutant algorithms

Posted on by

Back in 2014, I wrote two blogs (part 1 & part 2) about examinations and asked if they were fit for purpose. The conclusion – they provide students with a clear objective to work towards, the process is scalable and the resulting qualification is a transferable measure of competency. They are of course far from perfect, exams do not always test what is most needed or valued and when results are presented in league tables, they give a too simplistic measure of success.

However, I didn’t ask if examinations were fair, that is treating students equally without favouritism or discrimination.

In the last two weeks the question around fairness has been in the headlines following the government’s decision to cancel all A level and GCSE examinations in order to reduce the risk of spreading Covid-19. Whilst many agreed with this it did cause a problem, how could we fairly assess student performance without an examination?

Are examinations fair?

This is not a question about the fairness of an exam as a type of assessment, there are for example other ways of testing ability, course work, observations etc. Its asking if the system of which an examination is part treats all students equally, without bias.

In the world of assessment exams are not considered sufficiently well designed if they aren’t both reliable and valid. It might be interesting to use this as a framework to consider the fairness of the exam system.  

  • Validity – the extent to which it measures what it was designed to measure e.g. add 2+2 to assess mathematical ability.
  • Reliability – the extent to which it consistently and accurately measures learning. The test needs to give the same results when repeated. e.g. adding 2+2 is just as reliable as adding 2+3. The better students will get them both right and the weaker students both wrong.

The examining bodies will be very familiar with these requirements and have controls in place to ensure the questions they set are both valid and reliable. But even with sophisticated statistical controls, writing questions and producing an exam of the same standard over time is incredibly difficult.  Every year the same questions are asked, have students performed better or is it just grade inflation, were A levels in 1951 easier or harder than today? It’s the reliability of the process that is most questionable.

If we step away from the design of the exam to consider the broader process, there are more problems. Because there are several awarding bodies, AQA, OCR, Edexcel to name but three, students are by definition sitting different examinations. And although this is recognised and partly dealt with by adjusting the grade boundaries, it’s not possible to completely eliminate bias. It would be much better to have one single body setting the same exam for all students.

There is also the question of comparability between subjects, is for example A level maths the same as A level General studies? Research conducted by Durham University in 2006 concluded that a pupil would be likely to get a pass two grades higher in “softer” subjects than harder ones. They added that “from a moral perspective, it is clear this is unfair”. The implication being that students could miss out on university because they have chosen a harder subject.

In summary, exams are not fair, there is bias and we haven’t even mentioned the impact of the school you go to or the increased chances of success the private sector can offer. However, many of these issues have been known for some time and a considerable amount effort goes into trying to resolve them. Examinations also have one other big advantage, they are accepted and to a certain extent the trusted norm and as long as you don’t look too closely, they work or at least appear to. Kylie might be right, “it’s better the devil you know”….. than the devil you don’t.

The mutant algorithm

Boris Johnson is well known for his descriptive language, this time suggesting that the A level problem was the result of a mutant algorithm. But it was left to Gavin Williamson the Secretary of State for Education to make the announcement that the government’s planned method of allocating grades would need to change.

“We now believe it is better to offer young people and parents’ certainty by moving to teacher assessed grades for both A and AS level and GCSE results”

The government has come in for a lot of criticism and even their most ardent supporters can’t claim that this was handled well.

But was it ever going to be possible to replace an exam with something that everyone would think fair?

Clarification on grading

To help answer this question we should start with an understanding of the different methods of assessing performance.

  1. Predicted Grades (PG) – predicted by the school based on what they believe the individual is likely to achieve in positive circumstances. They are used by universities and colleges as part of the admissions process. There is no detailed official guidance as to how these should be calculated and in general are overestimated. Research from UCL showed that the vast majority, that is 75% of grades were over-predicted.
  2. Centre Assessed Grades (CAG) – These are the grades which schools and colleges believed students were most likely to achieve, if the exams hadn’t gone ahead. They were the original data source for Ofqual’s algorithm. It was based on a range of evidence including mock exams, non-exam assessment, homework assignments and any other record of student performance over the course of study.  In addition, a rank order of all students within each grade for every subject was produced in order to provide a relative measure. These are now also being referred to as Teacher Assessed Grades (TAG)
  3. Calculated grades (CG) – an important difference is that these are referred to as “calculated” rather than predicted! These are the grades awarded based on Ofqual’s algorithm. They use the CAG’s but adjusts them to ensure they are more in line with prior year performance from that school. It is this that creates one of the main problems with the algorithm…

it effectively locks the performance of an individual student this year into the performance of students from the same school over the previous three years.

Ofqual claimed that if this standardisation had not taken place, we would have seen the percentage of A* grades at A-levels go up from 7.7 % in 2019 to 13.9 % this year. The overall impact was that the algorithm downgraded 39 % of the A-level grades predicted by teachers using their CAG’s. Click here to read more about how the grading works.

Following the outcry by students and teachers Gavin Williamson announced on the 17th of August that the Calculated Grades would no longer be used, instead the Centres Assessed Grades would form the basis for assessing student performance.  But was this any fairer, well maybe a little, but it almost certainly resulted in some students getting higher grades than they should whilst others received lower, and that’s not fair.

Better the devil you know

The Government could certainly have improved the way these changes were communicated and having developed a method of allocating grades scenario stress tested their proposal. Changing their mind so quickly at the first sign of criticism suggests they had not done this. It has also left the public and students with a belief that algorithms dont work or at the very least should not to be trusted.

Perhaps the easiest thing to have done would have been to get all the students to sit the exam in September or October. The Universities would then have started in January, effectively everything would move by three months, and no one would have complained about that would they?

Double entry bookkeeping replaced by internet

Posted on by

There is an interesting question being asked at the moment, given that fact-based knowledge is so accessible using the internet, is there a case for not teaching facts at all?

According to Don Tapscott, a consultant and speaker, who specialises in organisations and technology, memorising facts and figures is a waste of time because such information is readily available. It would be far better to teach students to think creatively so that they can learn to interpret and apply the knowledge they discover online.

“Teachers are no longer the fountain of knowledge, the internet is”
Don Tapscott

Is this the solution for educators with an over full curriculum, the result of having to continually add new content to ensure their qualification remains relevant and topical? Perhaps they can remove facts and focus on skills development? After all its skills that matter, knowing is useful but it’s the ability to apply that really matters …right?

What makes you an accountant

When you start to learn about finance, you will be taught a number of underpinning foundational subjects including, law, economics, costing and of course basic accounting. Sat stubbornly within the accounting section will be double entry bookkeeping. This axiom is fiercely protected by the finance community such that if anyone questions its value or challenges its relevance they will be met with pure contempt. And yet, is the knowledge as to how you move numbers around following a hugely simple rule i.e. put a number on one side and an equivalent on the other of any use in a world where most accounting is performed by computers and sophisticated algorithms? I am sure there will be similar examples from other professions and industries. The challenge being, do doctors really need to understand basic anatomy or lawyers read cases dating back to 1892?

“Everyone is entitled to his own opinion, but not to his own facts”
Daniel Patrick Moynihan

But Knowledge is power

Daniel T. Willingham is a psychologist at the University of Virginia and the author of a number of books including, why students don’t like school. His early research was on the brain, learning and memory but more recently he has focused on the application of cognitive psychology in K-16 education.

Willingham argues that knowledge is not only cumulative, it grows exponentially. In addition, factual knowledge enhances cognitive processes like problem solving and reasoning. How knowledge Helps.

Knowledge is cumulative – the more you know the more you can learn. Individual chunks of knowledge will stick to new knowledge because what you already know provides context and so aids comprehension. For example, knowing the definition of a bond ‘a fixed income instrument that represents a loan made by an investor to a borrower (prior knowledge), enables the student to grasp the idea that anything fixed has to be paid by the company (the lender) regardless of its profitability and this is the reason debt is considered risky. (new knowledge)

Knowledge helps you remember – the elaboration effect has featured in a previous blog. In essence it suggests that the brain finds it easier to remember something if it can be associated with existing information. Using the same example from above, it is easier to remember that bonds are risky if you already knew what a bond was.

Knowledge improves thinking – there are two reasons for this, firstly it helps with problem solving. Imagine you have a problem to solve, if you don’t have sufficient background knowledge, understanding the problem can consume most of your working memory leaving no space for you to consider solutions. This argument is based on the understanding that we have limited capacity in working memory (magic number 7) and so to occupy it with grasping the problem at best slows down the problem-solving process, but at worse might result in walking away with no solution. Secondly knowledge helps speed up problem solving and thinking. People with prior knowledge are better at drawing analogies as they gain experience in a domain. Research by Bruce Burns in 2004 compared the performance of top chess players at normal and blitz tournaments. He found that what was making some players better than others is differences in the speed of recognition, not faster processing skills. Players who had knowledge of prior games where far quicker in coming up with moves than those who were effectively solving the problem from first principle. Chess speed at least has a lot to do with the brain recognising pre learned patterns.

Skills are domain specific – not transferable

There is one other important lesson from an understanding of knowledge – skills are domain specific. The implication being that teaching “transferable skills” e.g. skills that can be used in different areas, communication, critical thinking etc doesn’t work. A skill (Merriam Webster) is the ability to use one’s knowledge effectively and readily in execution or performance. The argument being that in order to use knowledge effectively, it needs to be in a specific domain.
In July 2016 the Education Endowment Foundation in the UK released the results of a two-year study involving almost 100 schools that wanted to find out if playing chess would improve maths. The hypothesis was that the logical and systematic processes involved in being a good chess player would help students better understand maths i.e. the skills would transfer. The conclusion however found there were no significant differences in mathematical achievement between those having regular chess classes and the control group.

Long live double entry bookkeeping

This is an interesting topic and open to some degree of interpretation and debate but it highlights the difficult path curriculum designers have to tread when it comes to removing the old to make space for the new. In addition there is a strong argument to suggest that core principles and foundational knowledge are essential prerequisites for efficient learning.
But whatever happens, we need to keep double entry bookkeeping, not because knowing that every debit has a credit is important but it helps structure a way of thinking and problem solving that has enabled finance professional to navigate significant complexity and change since Luca Pacioli allegedly invented it in 1494.

And the case from 1893 – Carlill v Carbolic Smoke Ball Company

Sensemaking, humility and the humanities

Posted on by

human-being-girl-picture

For a variety of reasons, I have been engaged this month in thinking not so much about examinations but what subjects should be examined.

Whilst the news has been dominated by terrorism, Trump and Brexit, we may be facing a far bigger problem, of which these news stories are a good example, how can we be sure of making the right decisions in a world of mass information, complexity and change.

People voted Brexit for a whole variety of reasons, many “facts” were presented in simple terms, we will save £350m a week and this money will go into the NHS, immigration will be reduced as we gain control over our borders. Yet these facts are far too simplistic, any level of analysis, critical thinking and challenge would have revealed the difficulty of delivering them, and in many instances they won’t be delivered. If this is the case, did people vote to leave, or stay not on the facts as presented but using other criteria, maybe they were just naive and placed far too much trust in Politicians or perhaps they had never been taught about sensemaking, humility or studied the humanities.

Sensemaking

An interesting article caught my eye earlier in the month, “Silicon Valley needs to get schooled”. it was by Christian Madsbjeg, author of the book Sensemaking and senior partner in ReD, a strategy consulting company based on the human sciences. In the article Madsbjeg argues that the reason for a lack of new and exciting products from Silicon Valley is not because of a shortage of ideas but a complete failure to understand people.

In the book Sensemaking he expands on the problem. In order to cope with complexity, we look to science, logic and the algorithm (a rules based process) for a solution. On the face of it crunching big data so that it spews out the correct answer seems perfect, but, and this is a quote from the book, Madsbjeg makes a very important point, he says “we stop seeing numbers and models as a representation of the world and we start to see them as the truth – the only truth”.  We are in fact looking at the numbers without the context of the world from which they came or a sufficiently deep understanding of the behaviours we are measuring.

We rely on science and the scientific method for so much of what we do but where people are involved we need a different approach. To put it another way “When human beings enter the equation, things go non-linear” Neil deGrasse.

Sensemaking is “how we make sense of the world so we can perform better in it”. It recognises that situations are complex and information ambiguous. It requires people to make a continuous effort to understand the connectivity that exists between people, places, and events in order to anticipate their trajectories and act accordingly.

Humility

trumpwillwin-notextIntellectual humility as defined by the authors of a recent paper entitled, Cognitive and Interpersonal Features of Intellectual Humility is the opposite of intellectual arrogance or conceit. It is in effect, recognising that you could be wrong. One of the findings from the research was that people who displayed intellectual humility were better than the control group at evaluating the quality of evidence they had been presented with. A very useful skill indeed, given the world of false news in which we currently find ourselves.

Humanities

And what job will you get after studying History for three years……

The humanities (English, History, Philosophy etc) have been given a bad press in recent times. Overshadowed by the drive to develop coding skills and with the constant chanting of STEM (Science, Technology, Engineering and Mathematics) in the background, it’s not surprising that less people are studying them. They were at an all-time low in 2014 at 6.1% of all bachelor’s degrees, a long way of the 1967 record of 17.2%.

But it is generally recognised that the humanities can teach us a lot. In another reference from Christian Madsbjeg’s book, Sensemaking, he suggests the humanities can teach us, one that other worlds exist, two that they are different and three, we learn how to imagine other worlds that in turn helps us better understand our own.

As with sensemaking and humility, are these not the types of skills we need to learn?

Examinations – what to examine?

What subjects should be examined depends to a large extent on what job you would like to do. But with the claim that 60% of 11 year olds will leave school to do jobs which have not yet been invented it’s hard to know the answer. What we do know is that the world is unlikely to slow down, change not happen, data become less available and complexity give way to simplicity. As a result, we need to teach people and so examine the skills that will help them better navigate this world. Maybe when those primary school children go onto higher education they will be studying sensemaking, humility and the humanities.

Even though the ink is barely dry on the letter sent by Theresa May bringing about our formal negotiations to exit Europe, the interesting thing is we will never know if this was a good or bad decision. Because post Brexit people will behave differently, some will work hard to make the impossible possible whilst others will continue to frustrate the process, and none of that could have been foreseen at the time.

So, let’s hope the basis for the original decision to leave was not because of the headline – We will save £350m a week and this money will go into the NHS!

Teaching to the test – another point of view

Posted on by

point-of-veiw-2A point of view is a programme on radio 4 that allows certain well-read, highly educated individuals, usually with large vocabularies to express an opinion. It lasts 10 minutes and is often thought provoking, concluding with a rhetorical question that has no answer.

This week Will Self the novelist and journalist gave his point of view on teaching to the test, as you might imagine it caught my attention. Self starts by telling a story about the life of a “good student,” and how it would unfold. He describes the way in which their concentration intensifies when the teacher states that what they are going to learn next is important and often examined. The story continues, as a result of their diligence and technique, the “good student” gets the necessary grades to go to University. They don’t however select the University on the basis of the course of study or on what they passionately wish to learn, no its based on the Universities credibility in league tables.

Upon successfully gaining a degree the student, now an employee gets a desk job that rewards a similar style of rubric mentality. As an employee, they are assessed against targets, performing well only on the ones that promise promotion and a pay rise. Eventually they retire and die.

Self concludes that this ordinary, dull, uninspiring life started back in the classroom all those years ago, when the teacher failed to educate and inspire, and simply taught to the test.

Over egging the pudding

There is a logic to this story, and it sounds ever the more inevitable as Self narrates it in his black and grey voice. But that’s all it is, a story. It avoids detail and colour, offering little regard as to the individual’s ability to reflect at some point in their life and ask searching and probing questions. It is as if somehow because the teacher highlighted the importance of one piece of knowledge it somehow stifled the student’s capacity to one day think for themselves.  Self is how they say, overegging the pudding, taking an interesting question as to the impact teaching to the test might have and serving up an omelette.

Teaching to the test is not bad

Brunel university asked a question as to what makes an unmissable lecture. In addition to many arguably more commendable answers, including the passion of the tutor and because they wanted to learn, the likelihood of the subject being taught having a high probability of being in the exam was key. Suggesting that a specific topic might be on the exam paper firstly, ensured a good attendance and secondly guaranteed the student listened intently.

Attention is important but even for the diligent student focus is vital. Learning everything is simply not possible, faced with 20 chapters, the student needs some clue as to where they need to direct their energy and time. Of course, the educationist will say that everything is important, but saying that will not make it so. Knowing that something is examinable at least gives a starting point and helps guide the student through the material quickly and efficiently. It’s also worth adding that It does not exclude the need to be inquisitive, in fact by making the student read a particular topic it may inspire them to find out more.

Exams and exam answers also provide examples of what is expected and the standard the student must reach if they are to be successful, no amount of narrative in the student handbook or curriculum guidance will do this as effectively.

The type of assessment matters

Of course, in Selfs world, teaching to the test removes the need to do anything more than learn about what will be in the exam. He suggests that students need to think outside the box rather than simply tick them. I have to admit I like that sentence.

But he does have a point, if the test is so narrow that it only assesses memory or a very small part of the syllabus then that is all the student will focus on. But that is just a bad test, this is of course where I am in danger of becoming idealistic and painting a picture that is not a true reflection as to what is happening. Not all tests are good, and undoubtedly some students will pass with limited thought and little more than good memory skills. Yet with changes in technology it becomes ever more possible to build tests and simulations that asses the student ability to perform in real world situations, and for that matter think outside the box.

Teaching to the test has become a term used to describe bad teaching and poor assessment and no one would agree that either of these are desirable. But it is not the process that’s problematic, it’s the application. Testing in its many forms is part of learning but it needs done well and thoughtfully.

In conclusion

Having now read the blog I would encourage you to listen to Will Self – click. It is of course not for me to say who presents the right point of view, you need to make up your own mind. For those however who were taught to the test no matter how long ago, you probably won’t understand even what I am asking because to the best of my knowledge this question has never been tested before……………?

 

Test obsession and Test Anxiety

Posted on by

Tests anxiety

“We live in a test conscious and test giving culture in which the lives of people are in part determined by their test performance”

Sarason, Davidson, & Lighthall

What’s interesting about this statement is, it was first published in 1960 and was based on students in the US, yet would not seem out of place in describing the situation in the UK.  The UK, as with so many other things has unfortunately caught up with the US and become a nation that tests and measures…everything.

Where a person’s worth is judged only by the tests they have passed it is perhaps not surprising that examination success has become so important and test anxiety increased.

But it’s not just the UK, this is a global obsession, take China for example where the pressure to succeed has become so intense that cheating in the Gaokao, the nation’s A-Chinese-invigilator-sca-010university entrance exam is a major problem. The government has not been slow to react and for the first time anyone found cheating will face a possible seven year jail sentence. In Ruijin, east China’s Jiangxi Province, invigilators use instruments to scan students’ shoes before they entered the exam hall, while devices to block wireless signals are also used to reduce the opportunity to cheat.

Test anxiety or stress

Stress is a broad term that is experienced when you find yourself in adverse or demanding circumstances, sitting an exam perhaps. Test anxiety is a situation specific type of stress, experienced by people who find examinations threatening. Recently, there has been an increased interest in exam stress and test anxiety in the UK and a need for it to be given closer academic scrutiny.

The research so far shows that test anxiety can actually impair learning and hurt test performance. And this is the issue, are students underperforming in examinations, which as stated above can have a significant impact on their lives not because of their lack of knowledge or even their ability to apply knowledge, but simply because the medium used to assess them is an exam.

In simple terms test anxiety effects exams results and exam results play a major part in people’s lives.

There are three components of test anxiety (Zeidner 1998)

  • Cognitive – the negative thoughts you can have during tests e.g. “if I fail this I will fail all my examinations” and the performance limiting difficulties experienced as a result of anxiety e.g. inability to read questions clearly or solve problems.
  • Affective – physical symptoms e.g. trembling, tension etc.
  • Behavioural – test anxiety creates an environment that encourages students to avoid studying or best delay it.

The reason people develop test anxiety is thought to be rooted in certain social issues e.g. how you are judged by others and the fear of failure in the public domain. It may also be related to the type of anxiety people experience when they have to make a best man’s speech, for example. Another aspect is that it is not always what others think, but what you think of yourself that is the issue and so the expectation of exam failure could impact on an individual’s ego and self-esteem.

Phase one is OK BUT

I think in the UK we are through the worst part of this, let’s call it phase one, and by that I mean we know that examinations and testing are not the answer, and that people are not their exam result. We have learned this the hard way by producing groups of exam qualified students, releasing them into the world of work, ill prepared to cope with the demands of the workplace. In addition, we have developed helpful techniques that enable people to better cope with test anxiety. Some of these I have discussed in previous blogs, Stress or Pressure – Don’t let the bridge collapse, Exam stress – Mindfulness and the “7/11” to name but two.

BUT………we still have some way to go with phase two, which involves answering the question, what do we replace exams with if they are so bad? And until we solve that, helping good people perform in the system we have just now is the best we can do.

 

Currywurst , apples and the £33,000 a year boarding school

Posted on by

currywurst

Currywurst fast becoming the most popular dish in Berlin

I have three relatively unrelated stories this month.

Berlin

The first follows a trip to Berlin this weekend and the obligatory city tour. Just of Unter den Linden you will find a sunken glass plate between the pavement that provides a view into a room full of empty bookshelves, sufficient to house the 20,000 books that were destroyed on this site on the 10th of May 1933. Some 70,000 people, many German university students gathered to burn books with “unGerman” ideas. Joseph Goebbels joined the students at the bonfire and declared: ” The future German man will not just be a man of books, but a man of character.” “It is to this end that we want to educate you.”

This was not only symbolic but practical. Without these books or to be precise only having the ones that remained, it might just be possible to create a nation of people who all think the same? Who all share the same ideals and live life by the same values. And yet of course this is not how you create an educated society, a society able to think for itself, the result will be compliance and idealism. Which was more likely what Goebbels had in mind.

Goodwill Hunting

When I got home one of my favourite movies had already started. Good Will Hunting tells the story of a maths genius, Will Hunting who struggles to come to terms with his past and as a result is unable to make a commitment for fear of being let down. But for me the best scene is where Will confronts a first year grad student who has been belittling his friend Chuckie.

Will: “Wood drastically underestimates the impact of social distinctions predicated upon wealth, especially inherited wealth…” You got that from Vickers. “Work in Essex County,” Page 98, right? Yeah I read that too. Were you gonna plagiarize the whole thing for us, you have any thoughts of your own on this matter? Or is that your thing, you come into a bar, you read some obscure passage and then you pretend, you pawn it off as your own, your own idea just to impress some girls? Embarrass my friend?

Will: See the sad thing about a guy like you, is in about 50 years you’re gonna start doin’ some thinkin’ on your own and you’re gonna come up with the fact that there are two certainties in life. One, don’t do that. And two, you dropped a hundred and fifty grand on a fxxxin’ education you coulda’ got for a dollar fifty in late charges at the Public Library.

Clark: Yeah, but I will have a degree, and you’ll be serving my kids fries at a drive-thru on our way to a skiing trip. Will: [smiles] Yeah, maybe. But at least I won’t be unoriginal.

This of course challenges the idea that knowledge is the same as thinking. Knowing something is not the same as having manipulated the information in your head, questioned and explored the arguments from many angles, the result should be an opinion, not someone else’s but your own.

Wellington college

And finally to bring my trilogy up to date. Only this week Wellington College who boast George Orwell and Sebastian Faulks as past students announced that have introduced a six-hour interview process to help teachers see beyond children’s over tutoring. They encourage families to spend more quality time instead of having their children tutored, Julian Thomas, Wellington’s head master, said: “When parents ask how should they prepare their children for our selection process, we tell them ‘have supper together and talk as a family; go to plays and good films and discuss them. Help them to think critically about the world around them; enjoy interesting conversations go out for walks and see the world”.

What Wellington are trying to do is encourage behaviours that will help children become thinking adults.

In conclusion

Knowledge like information is becoming increasingly accessible, unlike Germany in 1933 we have no restrictions, the internet has given us that power. But learning is more than knowledge you have to take the facts and make them your own, stitching them together carefully and thoughtfully, and that takes time and a considerable amount of effort. On the whole absorbing information will probably not come as easy to you as it did to Will Hunting, yet even with his genius he only knew what he had read. His learning had in fact only just begun as he went looking for the real life experiences that would test if he really understood anything at all.

Examinations are moving more towards tests that require you to think, asking that you provide an opinion or make recommendations given a set of circumstances, case studies are a good example. Of course you need knowledge as a foundation but the ability to sift quickly through the vast amount of information available may in time become more important than knowledge itself .

Ps apples is taken from Good Will Hunting – “How ya like those apples”

 

 

The tip of the iceberg – exam tipping is becoming obsolete

Posted on by

tip1

Assessment is changing, there was a time when all examinations were sat in a room, the answers would be hand written on a piece of paper and a retired English teacher would stand at the front reading out instructions as to what you could and couldn’t do in the next three hours.

Not any more…….you request a date that is convenient, turn up at the exam center, no longer is this a sports hall, it might be a driving school test center or the college you studied at. Then you log onto the PC and answer questions on the computer screen in front of you. The results may be immediate; it depends on if it is “human marked” or computer marked.

But in some ways these changes are only the tip of the iceberg!

What no past exam papers.

As examinations move into the digital world we are seeing other changes as well. There is a move towards objective testing, scaled scoring and examining bodies no longer providing past exam papers, what did you say, no past exam papers……!

This is partly down to the nature of the test i.e. you can’t provide an exact replica of a past exam question if it is an objective test. Remember objective test questions are randomly selected from a pool, and are different for each student. But there is also a shift towards some examining bodies only providing an example of the type of questions that could be set rather that a continuous flow of, the last exam papers.

If the test changes – how you study (and teach) has to change

Now for someone who has advocated that students analyse past exam questions in order to identify key areas so as to better direct their studies, this is a bit of a blow. It has also been the method I have used in the past to focus my own delivery in class and on line. Of course using past exam questions has always been much more than just spotting key areas, it is about focus, providing a place to start, showing content in the right context, helping with writing style etc.

There will still be past questions, sample questions will be provided. What we don’t know is how representative they will be of the examination. Or will it be as we have seen in the past with pilot and specimen papers, they change over time, drifting away from the original in terms of style and emphasis. Although I can see the logic in examining bodies not releasing papers, I hope they will continue to keep the sample papers fresh, in keeping with current thinking about the subject and how it will be examined.

What to do?

Students and tutors still need focus, there has to be emphases on key areas in order to chunk the content so that it can be more easily learned, it’s just that we won’t be able to use past questions or at least as much as we have in the past. That emphasis will now have to come from articles written by the examiners, examiner reports and syllabus weightings. If faced with a new subject where there is only one sample paper, it will be necessary to read the guidance from the body closely, noting reference to “this being a key part of the subject” or “one the examiner thought was answered badly in the past.” These together with the syllabus weightings and specific learning outcomes will have to be your guide. It is of course possible that the subject has not changed much from before and so some of the older past question can be used. As far as questions style is concerned then that will have to come from the questions and answers that are published, it may not be ideal but it’s the best we can do.

The overall impact of these changes is that students will have to know more, something that is hard to argue with. Students and tutors alike will have to devote far more time to the subject, which is fine if students have the time and can afford the extra costs involved in longer periods of study.

But it’s not all bad news, new technologies can help students make the most of dead time, studying on the train using their mobile phone for example. Also knowledge is more freely available than ever before as many top institutions provide a huge amount of free easy to access content online.

One final thought, examinations may change and they may not be fair but on the whole they are equal, everyone as before is in the same boat, and someone will always pass, wont they!

The future – Sitting the exam at home?

On line exams

An online student, all be it a mature one shows his ID to the online assessor

And maybe even the exam room will become obsolete. Proctur U is a US based company that also has a presence in the UK offering online invigilation. Watch this video to see how it works and judge for yourself

 

 

Video killed my teacher – metaphorically speaking

Posted on by

Video killed the radio star

What did you do the last time you needed to repair, cook or dare I say learn something? Did you google it and follow the link to YouTube? If so you are no different to the over one billion people who actively use YouTube every month.

This blog is not actually about YouTube but the medium of video and the increasingly important role it plays in our daily life and how we use it to learn.

 

 

Social learning and the bobo doll

Albert Bandura is the Professor Emeritus at Stanford University and is widely regarded as one of the greatest living psychologists. He is perhaps most famous for his Social learning theory, the theory of how people learn by observing others, and the bobo doll experiment. Click here to listen to Bandura himself explain.

Behavioral theories of learning suggested that all learning was the result of associations formed by conditioning, reinforcement, and even punishment, see Skinner and Pavlov. Bandura’s social learning theory proposed that learning can also occur simply by observing the actions of others. And that is where the true value of video lies, it is in the ability of people to watch what others do and learn from them.

What makes a good learning video?

Firstly, as with any form of delivery it needs to planned and structured. What is the purpose of the video, why use video and not send an email? Think about the audience, why will they want to watch it, what makes it relevant for them? Break it into three sections, a beginning, where you tell the audience what you are going to tell them, the middle, where you actually tell them, and the end where you tell them what you have told them.

Secondly It has to be relatively short, 10 minutes is a maximum. Even 6 minutes of good video takes a lot of planning, equally it wont test concentration levels too much. This does not mean you can’t record many hours of video, it just needs to be chunked, labeled and structured so it can be easily followed.

And lastly think about your delivery. Pace, tone of voice and body language all help the learner. This is where you manage the mood of your audience, if your happy they will be happy. Generally, speak more slowly than you would normally but be careful toooo slowww can be boring, vary how you say something depending on what you’re saying. Also think about the visuals and if it would be better to show an image rather than talk. But don’t go mad and put too much on screen all at the same time, it gets confusing.

Examples of good video 

But of course the best way to explain the power of video in learning is to show the videos.

1.The queen of cooking Delia is also the expert of slow deliberate, perfectly planned presenting. Here she explains how to cook an omelet, notice the attention to detail.  Ps Delia left school at 16 without a single GCE O-level. 3.43 minutes in length.

2.Here is someone who breaks the presenting rules, certainly the one that says don’t talk too fast. However, CGP Grey is great at using visuals, his dialogue is fast but incredibly informative, its packed with information, and it’s funny. If you are confused by the US elections, you won’t be after watching this.  5.19 minutes in length.

3.Crash course is a little like Khan academy which I have written before, what makes it different is the humour and how it is shot to camera using powerful visuals. Watch this clip if you want to learn about supply and demand. This pushes the boundaries time wise at 10.21 minutes.

4.This is the big one certainly as far as hits are concerned. James Stevens, Vsauce, is watched by 19 million people. This one answers the questions as to, what would happen if everyone jumped at once? 7.12 minutes in length.

Why cramming works and making stuff up is okay

Posted on by

Will making stuff up

Will making stuff up

To a certain extent I have spent much of my career making things up. When I was a student that was not the case, I listened and learned and so when I spoke, I spoke with confidence that what I was saying was correct, because someone had just told me it was. Yet knowing is only the start, and in some ways a poor relative of the “figuring it out for yourself” technique.  I am reminded of quote from the film Good Will Hunting, which along with Dead Poets capture some really magical moments in learning.

Will Hunting – “See, the sad thing about a guy like you is in 50 years you’re gonna start doing some thinkin on your own and you’re gonna come up with the fact that there are two certainties in life.” “One, don’t do that.” “And Two, you dropped a hundred and fifty grand on a f***in education you coulda got for a dollah fifty in late charges at the public library.”

 Question practice – the secret to exam success

Having had no real formal educational training I have been exploring ideas as to why some techniques work and others don’t, why it is that student A passes yet student B who did exactly the same, failed. One clear observation from over twenty years in the high stakes exam world is that the most important activity that a student can engage in is, question practice. As a lecturer I would make statements, explain them using real world examples, get students to laugh, and maybe even enjoy the subject. But, the very best learning seemed to happen when the student was required to do a question. So it was with great interest that I read of some research that came out of the US in 2011, it’s called Retrieval Practice.

 Retrieval practice – the power of cramming

Retrieval practice is simply the process of retrieving something from memory.  So for example if I asked you, who was the Prime Minister that took us into the European Economic Community in 1973, you might say, on reflection Edward Heath. You already knew the answer but were forced to recall it. If however you were not sure who it was and were subsequently told (given feedback) it was Edward Heath and that Harold Wilson in 1975 held the first referendum, you are likely to remember both. But the most interesting and perhaps surprising aspect of this research is that not only can you recall the facts, it also leads to a deeper learning in so much that you can answer questions on related information. This in some ways gives credence to the idea that cramming information, maybe not at the last minute could be beneficial, not simply because you will remember it for a few hours’ but that it will lead to deeper learning.

Mark McDaniel is a psychology professor at Washington University in St. Louis

“We think of tests as a kind of dipstick that we insert into a student’s head, an indicator that tells us how high the level of knowledge has risen in there when in fact, every time a student calls up knowledge from memory, that memory changes.” “Its mental representation becomes stronger, more stable and more accessible.”

Jeffrey Karpicke, a professor of cognitive psychology at Purdue University

“Retrieving is the principal way learning happens.” “Recalling information we’ve already stored in memory is a more powerful learning event than storing that information in the first place,” he says. “Retrieval is ultimately the process that makes new memories stick.” “Not only does retrieval practice help students remember the specific information they retrieved, it also improves retention for related information that was not directly tested.”

Final thoughts

And so I am pleased to say that what I have observed in the classroom, that question practice improves exam results might be a little simplistic and that not only does it help students pass exams they might actually have been learning something at the same time 🙂

If you want to read more follow these links

To Really Learn, Quit Studying and Take a Test (New York Times)

Researchers Find That Frequent Tests Can Boost Learning (Scientific American)

Election special – who has had the best education?

Posted on by

The direction in which education starts a man will determine his future in life

Plato

uk-politiciansInspired by the election and the personalities that have been thrust upon us during the run up to next Thursdays vote, I thought it might be interesting to look at the academic backgrounds of those that have ambition to be Prime Minster. It might even give you some help in choosing who to vote for.

The contenders

8761f28.jpgDavid Cameron – born 1966 – David William Donald Cameron was bought up in Peasemore, Berkshire his father was a stockbroker and mother a retired Justice of the Peace. He attended Heatherdown Preparatory School (12 O levels – 4 A’s, 5 B’s, a C and 2 below C) before moving to Eton, the school that can boast 19 past British Prime Ministers. After A levels (3 A grades) he went to Brasenose College, Oxford, from which he graduated in 1988 with a first in philosophy, politics and economics.

 

Ed MilibandEd Miliband – born 1969 – Edward Samuel Miliband went to Haverstock Comprehensive School (Labours Eton), North London. His father, a Polish Jew, was one of the leading Marxist theorists of his generation. His mother, Marion Kozak, is a long-standing human rights campaigner. After completing his A Levels (A,A,B,B) he read Philosophy, Politics and Economics at Corpus Christi College, Oxford, gaining a 2:1 and later Economics at the London School of Economics where he received his MSc.

 

The King makers

Nick CleggNick Clegg – born 1967 – Nicholas William Peter Clegg was raised in Buckinghamshire by his Dutch mother, a special needs teacher and his half Russian father, Nicholas P Clegg CBE, the chairman of united trust bank. Clegg was educated at two independent schools, Caldecott School in Farnham Royal in South Buckinghamshire, and later at Westminster School in Central London. He went on to study Social Anthropology at Cambridge (2:1) and continued post graduate studies at University of Minnesota through a scholarship where he wrote a thesis on the political philosophy of the Deep Green movement. He then went to the College of Europe in Bruges for his Masters.

Nigel FarageNigel Farage – born 1964 – Nigel Paul Farage grew up in the Kent village of Downe. His father, Guy Justus Oscar Farage, was a stockbroker and heavy drinker left the family home when he was five. He was educated at Greenhayes School for Boys in West Wickham, then Eden park before joining  Dulwich College (The Dulwich experiment)  an independent public school in south London. After A levels he decided not to go to university, but to work in the City, trading commodities at the London Metal Exchange.

Nicola SturgeonNicola Sturgeon – born 1970 – Nicola Ferguson Sturgeon is the eldest of three daughters born to Robert Sturgeon, an electrician, and Joan Kerr Sturgeon a dental nurse. She grew up in Prestwick and Dreghorn and attended Dreghorn Primary School and then Greenwood Academy. She later studied at the University of Glasgow, where she read Law and graduated with a Bachelor of Laws (Hons) and a Diploma in Legal Practice.

 

And the winner is………

By most people’s standards all our candidates had a very good education, progressed academically and enjoyed school life. In addition, with the exception of Nigel Farage whose father left when he was 5, all have had full parental support.

Some statistics-In 1981 around the time David Cameron was taking his O levels only 25% of students got 5+ GCSE O levels, and around 10% went on to gain 3 A level passes. In 2014 (admittedly many years later – but it will give us an idea) Oxford received 17,000 applications for 3,200 places, that’s a 5% chance of being accepted.  So Cameron, Miliband and Clegg all stand out. Farage and Sturgeon are clearly well educated but do not have such a strong academic background.

Not a political statement but

Overall for me the person who has had the “best” education is Nick Clegg. David Cameron and to a lesser extent Ed Miliband have had a rather insular academic journey. But Clegg went to Cambridge, then travelled to the US and Europe to study, which must have broadened his views, exposing him to the opinions and cultures of other students, many with different backgrounds . And in second place I would probably go for Nigel Farage, yes I did say Nigel Farage, not the man you understand, but as a good A level student he went into the world of work, the university of life argument.

PS – The former Conservative Prime Minister John Major famously left his comprehensive school, Rutlish, in London with three O-Levels: history, English language and English literature.

 

The Price is right? – Value for Money Education

Posted on by

The Price is RightThe idea behind the price is right was that you had to guess the price of an everyday object. Not a complicated game I admit but by game show standards a successful one, it ran from 1984 to 2007. But how do you know if something is the right price? What does “right price” even mean?

 

Easier in a market where there are many similar products all providing a similar service or experience, not so easy when assessing the value is subjective, comparability difficult  and getting it wrong  expensive. This is exactly the situation you might find when trying to choose a course provider, a college or university. How do you know if you’re getting value for money, if one provider is more expensive than another is it extra profit, inefficiency or a measure of quality and so value?

What do you want?

The first question to ask is, what do you want from the course provider and how will you measure success? On the face of it the answer may seem obvious – it’s to pass the exam or get as high a grade as possible. But learning is about so much more than the exam result, isn’t it?

What about the skills you develop and the knowledge you acquire, what about the people you will meet and the inspiration, motivation and direction you will receive? These are difficult to measure and are often ignored yet in the long run are probably far more valuable than the passing of an exam.  Also would you be happy with knowing just enough to pass and then afterwards forgetting everything, is that what you pay for, is that value for money?

A high quality provider will teach content and explain concepts so that you not only retain knowledge but develop a deeper understanding of the subject matter.

So it’s all about passing?

You may want to pass but what method of study will give you the best chance of passing? How much contact time do you The price of everythingexpect? Are you sufficiently self motivated that you need little or no help, do you want the convenience of studying online or the discipline of having to go to lectures? You may be aware of the method you prefer but many students aren’t. The better provider will know enough about you as an individual, and if they don’t they will ask before making any recommendations as to what method of study is best.

Risk and cost of failure

How important to you is passing, or passing as soon as possible? It could be a false economy to go with the provider who suggests the course can be covered in 10 weeks at a cost of £1,000 when another suggests 15 weeks at £1,500. The longer course with the higher price could well be good value for money if you pass first time. Equally when the stakes are high you don’t want to make a mistake,  consider something like eye surgery, would you go to the cheapest provider where the risk of getting it wrong could be life changing.  Of course expensive does not always mean value for money. You will need to do some homework first. Below is some advice as to what to look for when choosing a provider before you part with any money.

What to look for in a course provider

Fundamentally it’s about trust and confidence in the provider. Education is not a commodity, it’s not homogenous, it’s personal and too important to get wrong

Here are a few things to look out for and questions to ask.

  • Experience and quality of the Teachers/Lecturers – having a stable and experienced lecture team is an indication of quality. Ask how the college ensures their staff are up to date, do they have a formal training scheme? What research credentials do they have?
  • Long term player – How long has the organisation been in existence, ask them what their long term strategy is for learning or at least what they think the future might hold.
  • Where do they rank in league tables – maybe they have industry awards or accreditation by external bodies.
  • Investment in the future and level of innovation – what do the premises look like, are they well maintained? What technologies have they introduced recently?
  • What are the range of different study options (length of course/F2F contact time etc) and levels of personalisation – for you to have the best course the provider should be able to offer some degree of personalised learning.
  • Ask if you can try before you buy – What have you got to lose they can only say no. Oh and ask them how easy it is to transfer to other courses and get your money back if you’re not happy.
  • And finally one of the most useful ways in making any decision is to ask friends/colleagues what they think or experiences have been, and don’t forget to check them out on the social media sites.

Conclusion

There is a lot more to this debate and the topic is certainly worthy of another blog. Value for money is a big question in education. I have not for example even mentioned the cost of education in the context of employability and student debt. Nor which subjects have the highest employability statistics etc.

The purpose of this blog was to highlight the complexity of choosing a provider and to give some advice as to what to look for.

In summary, clarify exactly what you want from your course provider before you start looking, ask some of the questions above and dependant on the answers you get make your decision. And if all goes to plan you will you end up with the right provider, at the right price and so great value for money.

PS Happy New Year everyone – I think as far as learning and exams are concerned 2015 is going to be as interesting and as uncertain as 2014. I am looking forward to it.

 

Are exams fit for purpose (part two) – what are the alternatives?

Posted on by

You dont fatten pigs 2

Last month’s blog came to the conclusion that examinations* are fit for purpose or at least “a purpose.”

They provide the student with a clear objective to which they can direct their efforts and focus attention and are a transferable measure of competency that can be assessed at scale. The “at scale” point is important as there are many ways of assessing competence but few that can cope with the need to test thousands of students all at the same time.

The main problem with examinations is that they don’t always examine what is most valued; the method of assessment often has significant limitations as to what it actually tests and the results are presented in league tables that give a far too simplistic view of success.

I am not sure we can resolve all of these but it might be worth exploring other options, specifically alternative methods of assessment. For example If you change the method of assessment from a formal, often timed written exam to say a portfolio of work, not only do you change the method of assessment but you will change what is being examined, two birds with one stone perhaps.

Different methods of assessing competence

Open book exams

Open book assessment offers a way of testing application rather than memory. Students have access to a text book that contains information relevant to what they are being asked. It’s the use of knowledge that is important, not the knowledge itself. The idea of open book could easily be adapted, why not allow students access to the internet during the exam, they could look up anything they wanted. Is this not more representative of what happens in the real world?

Take out exams

Similar to the above the so called “take out exam” allows the student to take the exam away to work on at home using whatever resources they prefer, books, internet etc. They return the next day with a completed answer. This can work better than you might at first think so long as you have a robust mechanism to detect plagiarism. There are several very good software packages that can spot the most sophisticated types of copying.

Case studies/simulations

A case study provides an environment for the student to demonstrate they can use their knowledge to solve problems and or offer advice in a virtual world. Most case studies tend to be written but this is one area that we could see some clever and affordable use of technology to better simulate the real world.

Performance tests

In a performance test students are required to demonstrate a skill/process, create a product etc while being observed by the assessor who will evaluate the performance. A great example of testing ability to apply knowledge but suffers from the subjectivity of the assessor and has limited application at scale.

Portfolios

Portfolios are most often collections of the student’s work that demonstrate their ability to perform a specific task. These can be simulations of the real world or portfolios of work actually undertaken on the job. A portfolio can include written documents, emails, audio or video recordings, in fact anything that provides evidence as required by the assessor.   Portfolios are perfect for assessing application but the process of assessment is expensive and not without bias.

Viva Vocal – (living voice) Oral exam

Often used to test PhD students, an oral exam gives the assessor chance to question the student. This is a very effective method where you are looking for higher level skill and depth of understanding. As identified last month it’s probably one of the oldest forms of assessment.

Digital badges – capturing the learning path

Being awarded a badge as recognition of achievement is something many will be familiar with especially if you were a boy scout or girl guide. But digital badging is new and becoming increasingly popular because of the internet. A good example would be linkedin and the badges awarded to you by others as recognition of certain skills.  Many of the assessment methods above provide a first past the post type of assessment, you pass and that’s it. Digital badging on the other hand is a form of lifelong assessment that evolves along with your career.

Digital badging for me is one of the most exiting forms of assessment and I am not alone Nasa have been using digital badging since 2011. Read more about digital badging.

Assessment in the future

Scanning for competenceThe list above is far from comprehensive and many other equally valid types of assessment exist e.g.  Role plays, Slide presentations, Assignments etc but what might assessment look like 15 years from now. Well how about using MRI scans to identify which parts of the brain are being used?  Not sure it will catch on but it would provide some interesting evidence as to how the student is getting to the answer, simple memory or a genuine and deep understanding .

*Examinations defined as a written test administered to assess someone’s level of understanding, knowledge or skills

Are exams fit for purpose? (part one)

Posted on by

take-the-same-testI have written in the past about what passing an exam proves but have never questioned if exams achieve what they were originally designed to do, are they fit for purpose?

Firstly let me define what I mean by an exam. A written test administered to assess someone’s level of understanding, knowledge or skill that results in a qualification if successful. This is in contrast to a test which is a method of assessing someone’s level of understanding, knowledge or skill often as part of a course in order to provide feedback. A test does not have to be written. Although exams don’t have to be written either, many are and initially at least I would like to keep the definition as narrow as possible.

In order to answer the question, are exams fit for purpose we must first take a step back and look at how we got to where we are now.

 

A brief history of examinations

The first standardised test is believed to have been introduced by the Chinese in 606 AD to help select candidates for specific governmental positions. However most examinations around this time would have been oral, requiring the candidate to recite a dissertation or answer questions. Although there is evidence of written exams being used as early as 1560*, it was not until the 1820’s that many Universities began to adopt the practice. From 1850 onwards the written exam became the norm in most UK Universities. In 1854 under the Gladstone government selection of Civil servants was based on their ability to pass an exam, this time however it was written.

Bureaucracy – In 1917 to help bring some order to what had been described as chaotic the Certificate and the Higher School Certificate were introduced. Then in 1951 we had the General Certificate of Education (GCE) examinations, more commonly known as Ordinary ‘O’ level and Advanced ‘A’ level , these were normally taken at 16 and 18.

In the 1960’s the CSE (Certificate of Secondary Education) was born, opening up qualifications for all, not just those that went to Grammar school. However this two tier system was thought divisive and so in 1988 under the guidance of the then Education Secretary Sir Keith Joseph both sets of examinations were replaced by the GCSE. The GCSE was graded and contained credit for course work.  In 1991 the General National Vocational Qualifications, (GNVQS) were established intended to incorporate both academic and vocational elements, by 1995 these were accepted as ‘equivalent’ to GCSE.

In 2014 we find change again, gone is the course work and written examinations once again become the main method of assessment, although there will be grading, 1 to 9 with 9 being the higher mark. The exams will still be called GCSE’s, although officially they are known as GCSE (England). This is to avoid confusion with Wales and Northern Ireland, who are not changing.

Yes they are

Historically at least it would appear the purpose of the exam was to provide a recognised and transferable measure of competency in a given subject or discipline. The lack of transparency and consistency of the oral exam resulted in them being replaced with written ones and a more formal bureaucratic structure was developed to administer the process.

And in many ways there is very little wrong with this.

The problem is not with the exam itself, but with what is being examined. If as a society we value “thinking and creativity” for example, then should we not be examining these rather than subjects that require the candidate to do little more than rote learn facts.  Perhaps we should explore different methods of assessment, the written exam has its uses but hand written papers are looking increasingly outdated in a world that communicates electronically not only in short texts and tweets but with video and photos . In addition the way exam results are used in league tables to show winners and losers is divisive. It looks like a measure but has in fact become a target that schools and teachers must hit or be considered failures.

Please watch this it’s very funny…..and thought provoking

Not on the test

 

 

 

 

 

In the second blog about exams I want to look more closely at some of these points, in particular what other ways we can assess what people know.

*Assessment around this time was through debate between a number of learned people all at the same time and lasting for two hours or more.

Teaching to the test – Interesting research but the fat lady is still in good voice

Posted on by

Fat lady still singingThis week researchers from the University of East Anglia released some very interesting findings that resulted from testing 594 bio-science students in their first week of term at five universities.

The students selected would be considered by many more than competent in their subject, almost all had a grade A*, A or B in biology at A-level. Yet when they were given 50 minutes to answer 38 multiple choice questions on cells, genetics, biochemistry and physiology from their A level core syllabus, they only got 40% correct. The period of time between the students sitting their A levels and the test was three months.

Lead researcher for the study, Dr Harriet Jones, said: “What our research shows is that students are arriving at university with fantastic A-level grades, but having forgotten much of what they actually learned for their exams.”  She went on to say that the trend to teach to the test, to ensure good results for schools’ reputations, was the problem.

The schools are to blame then

The facts of the research are clear, students who had successfully passed a test, were unable to pass a similar test three months later. The conclusion reached is that the students did not understand (see my blog on understanding) their subject well enough and passed their A levels probably using little more than memory. And who is to blame, the schools of course, for teaching to the test. Why the school do this is worthy of further debate, but government pressure and the impact of league tables will certainly be in the mix.

But do employers not accuse Universities of delivering up similar ill prepared students. The test is different but from the employers perspective the result is the same. A University student who professes to know something but when tested “in the real world” doesn’t.

Does this mean that Universities are also teaching to the test!

It’s about the test etc

The problem is not in teaching to the test; the problem is with the test, the pass mark and possibly the marking. If the test was more Testing but for what!aligned to what the student needs to know/do at a fundamental level, the pass mark sufficiently high and the marker having some degree of autonomy to form judgements, then the results would probably be different. It could of course be that the exams are easier – Exam chief: ‘you don’t have to teach a lot’ for our tests.

The big criticism of teaching to the test is, it results in a narrowness of understanding, little in the way of depth and does not push students to think in abstract and creative ways. But if the test, which incidentally does not have to be in the exam hall or on paper/PC was able to “test” for these qualities then teaching towards it would perhaps be more acceptable.

Bottom line

Teaching to the test is unlikely to change, in fact given the popularity of league tables  in education just now it may well increase, but with more effective testing the results might be better students, happy Universities and even happier employers.

 

 

Eureka – I Understand Understanding!

Posted on by

I Understand!If you understand the subject you are studying your chances of passing the exam must be good.

A simple and perhaps obvious statement but what does understand mean and what do you have to do to truly understand something? Of course understanding is a key part of passing but it is not enough on its own, you can understand something yet fail because you run out of time, misinterpret the question, thought you understood but didn’t! etc.

To understand

The dictionary defines to understand as, to know what someone or something means, to grasp the meaning, to be familiar with, make sense of etc. Understanding is clearly different to knowing, for example, you may know that gravity is a force that pulls objects to earth but that does not mean you understand what gravity is or how it works. Of course you need both knowledge and understanding, the one is no good without the other. Examiners try to test for understanding by asking questions that require you to compare, contrast, explain, interpret etc.

Understanding is not a Eureka moment, it has different levels. It might seem that there is a point where you didn’t understand and then suddenly you did, a Eureka moment. In reality what you have done is move closer to gaining a better and fuller understanding. Ask any lecturer or teacher, often they will tell you they never fully understood something until they had to teach it, they just thought they did.

Proving you understand – The 6 facets of understanding

Understanding by design, Wiggins and McTighe (1998) is one part of an instructional design process that provides a very helpful framework we can use to explore the depth of understanding and perhaps more importantly what you can do to develop a deeper understanding. Think of it as a hierarchy with the easiest one first, the greater you’re understanding the higher the number.

1. Explain, the classic exam question – Explain to someone what the concept/idea means and say why. Explaining out loud to yourself or making a recording can be just as effective.

2. Interpretation – Relate the concept/idea to your own experiences, tell a meaningful story. Try to add something personal into your explanation. To do this you will need to reflect on past events, whilst attempting to find parallels with the concept/idea.

3. Application – Use the concept/idea in a different context. The ability to apply knowledge in different contexts (transfer) is a key milestone in learning as well as understanding. It should result in you never being caught out by a difficult exam question. Understand to this level and it doesn’t matter what the examiner asks.

4. Perspective – Read around the concept/idea, get other people’s views, and see the big picture. If your struggling with understanding, read another text book or my favourite is to go onto you tube and watch a video. The internet is great for discovering alternative views.

5. Empathy – Try to get inside another person’s feelings about the concept/idea. This is difficult as it requires you to put aside your feelings about the concept/idea and accept that it is not the only way of thinking about it.

6. Self Knowledge – Ask questions about your understanding, ask what are the limits of your understanding, what are your prejudices, become aware of what you don’t understand. Often called metacognition, the ability to think about thinking.

The Eureka moment

Understanding, like Eureka moments are not of course the result of sitting in a bath and suddenly finding you understand something you had previously found confusing. It is the gift of hard work and long hours of study, hopefully by trying some of the techniques above your depth of understanding will only improve.

Ps apparently the jeweller was trying to cheat the king….

Understanding by Design

Want to know more about understanding by design, watch this. 

 

 

50 Shades Darker – How much to test

Posted on by

How much to test

In August last year I wrote how Professional accountancy bodies believe that competency can be measured by a candidate scoring 50% and failure if scoring 49%. This all or nothing approach seems neither fair nor useful; hence the idea that grades of competence could be introduced e.g. 50% pass, 65% commendation, 75% distinction etc.

But the mark a student gets in an exam is only part of the story when it comes to measuring competence. Can a student be considered competent if the exam they pass only includes questions from say 75% of the syllabus? Yes the whole syllabus might be covered in an 18 month period but in any one exam 25% is not tested. Equally the 75% is often considered core and so examined every sitting, this means that a student need only focus on the 75%. Admittedly if the pass mark is 50% they need to score 50% out of 75% (67%) but with practice this is possible. One final observation, it is unlikely the student will score 0% on the non-core part of the syllabus. They may get say 5% or even 10% out of 25%. Not a great score but the 67% pass mark now becomes 58%. This logic sits at the heart of the exam driven approach.

Objective testing might be the answer?

Objective tests (OT) – test that are relatively short and can be unambiguously marked, are considered by some to be a weaker form of assessment, they are part of the dumbing down of examinations. The beauty of an OT question is that the marking is completely accurate, no marking bias at all. This is often ignored in traditional exams and not seen as a problem largely because it’s not that visible. But the OT does not solve the “how much to test” problem, in fact it makes it worse. If you are asking for less then you are examining less. So if the only benefit is the avoidance of marker bias why are more examining bodies using OT style exams, is it just about saving money…….?

An example

Imagine that you have 4,500 OT style questions that cover every aspect of the syllabus, let’s also assume that the student only has to answer 50. The 50 questions are randomly picked from the 4,500 in the question bank. Is it fair that a student is considered competent if they are only being tested on 1% of what they need to know?

I think the answer is Yes, because in order to be sufficiently prepared to answer 50 questions from a bank of 4,500 when there are no core topics you have to have be capable of getting all 50 correct (assuming a 100% pass mark) and because you don’t know which 50 are coming up this effectively means you have to be able to get all 4,500 correct. The examining body can of course control  the effectiveness and level of difficulty by changing the pass mark.

All OTs would be OTT

This is not an argument to suggest that all examinations should be assessed using OT type questions, they should not. For example they are not Less-is-Morevery effective at measuring a student’s ability to communicate or evaluate complex and ambiguous situations, but they should be considered part of the tool kit that examining bodies have in assessing competence.

So on the face of it OT’s may look like a soft option, anyone can tick a box but they are certainly not easy to get right. Maybe less is actually more…..

50 shades of Grade…..Measuring students worth

Posted on by

I don’t often write specifically about the market that I am most closely involved, that of accountancy training and education. But there does seem to be an anomaly in the way the accountancy world measure success that isn’t the case in many other professions and examinations.

To become a qualified accountant in the UK (ACCA, ICAEW,CIMA etc,  yes there are more) you have to pass a number of demanding examinations and submit evidence of practical experience. The exams are taken over 3 years and everything rests on a number of exams of 100 marks each, of which you need to get 50%. The pass rate for accountancy students varies from paper to paper but is around 60% In training and education terms these exams are often referred to as High stake exams. For some failure is not simply a setback in terms of time, it could result in a lost job.

50% good – 49% bad

The purpose of such exams (including the practical experience) is of course to ensure that those that pass are competent. But how competent….?  It seems that you are either competent or not, if you get 49% you are not competent, if you get 50% then you are. This makes the margin for error very high, many a career has taken a dramatic turn based on 1 mark. Now of course in any exam there has to be a point where someone succeeds and another fails but does competence not have many shades to it, is it not a continuous process rather than a discreet one.

Shades of Grade

You may be one of thousands of students (>300,000 actually) who have just had your A level results. But of course you don’t just pass or fail, in fact this has not been the case since 1963, even before then there was an indication of the mark. You are given a grade from A through to E and all are considered a pass, 98.1% passed their A levels this year.

And the grades refer to the marks e.g. E is 40%, a D is 50%, a C is 60%, a B is 70% and an A is 80%. A similar grading exists for Degrees, 2.1, 2.2 etc and for Law exams, Pass, Commendation, Distinction.

No pressure for excellence

So why is this not the case in accountancy? To be honest I am not sure. But by introducing grades above a pass you not only provide employers with a better indication of competency, you encourage colleges to teach to a higher and deeper level, motivated by the students who want to learn more. And that for me is the most important point; you create an incentive for students to try harder. At the moment if you are scoring 55% then why put in more work, a pass is a pass, but a distinction well that will make you stand out from the crowd.

Of course the market will adjust its perception of what a pass means, for example employers may only want students with a distinction rather than a simple pass, but they have always looked and will continue to look for differentiation, at least this is a meaningful one.

What it does not do is fix the 49% you fail problem, and careers will still hinge on 1 mark, but that unfortunately is the way with exams and the reason exam technique is so important.

So to all the accountancy examining bodies why not introduce grades – grey is good and isn’t it an accountants favourite colour?

Related articles

Tipping the exam is bad….right?

Posted on by
No TippingFor me holidays are not a treat, they are an essential way of  recharging your batteries and provide an opportunity to re think ideas and put things into perspective. I am writing this particular blog whilst visiting Washington DC for a few days before moving onto the real America, Disneyland!

 
 
 

Exam tipping
Now whether its because I was having to get my head around the whole US tipping culture I don’t know, but I found my mind wondered to a conversation I had just before I came away as to the merits of predicting topics that will be on the next exam, so called tipping. This is a fascinating question, and one that is at the heart of the exam focused approach. The logic of the exam focused approach is simple (see  earlier blog for more detail) if the main objective is to pass the exam then the best preparation is to practice The questions that will be on that exam. But on the basis we don’t have The actual exam, then the next best preparation is to practice questions that “you think” will be in that exam, hence the tip.
In many other training environments this approach is common and well accepted. My daughter was taken on the routes most used by the driving test examiner. Although on the day she didn’t know exactly which route it was going to be. As a result of the training and familiarity she felt well prepared, more relaxed which in turn built her confidence, and I am sure this improved her performance on the day.
So what’s the problem?
The problem is that if taken to an extreme, teachers might only teach what they think will be on that particular exam, the so called “teaching to the test”. This results in students not receiving a sufficiently broad or in-depth coverage of the subject and exposing them and their chances of passing to risk, heads you pass, tails you fail.
Application – Focus – Prioritisation 
However exam questions are great tools for application, focus and prioritisation. If a student is taught something and then set a question on that topic, they very quickly appreciate its relevance and value. It certainly helps with concentration, focus and  putting topics in order of importance. The, you can’t learn everything so stop worrying and get on with this question approach…..
Its all about time
If students have sufficient time they should of course learn everything. Although if you think about it you can’t learn “everything”, so there is always some degree of prioritisation.  But in the professional education sector, they don’t. Many hold down demanding jobs by day and study is part time. They look to the educator to give them some degree of priority in what to learn. This comes into sharp focus in the last few days before the exam, and that’s why they will ask for what is most likely to be on the exam, the tip.
So should the teacher tip
It’s all about degrees, at one end of the spectrum, it is very risky and narrows a students learning, at the other it gives focus and application in the last few days before the exam. Personally I would like to see  research on this area some solid evidence to focus the debate. There may be something out there but I have not read anything on this specific question.
But in the absence of such research my view is not to be too dismissive of tipping, it is very easy to completely write it off, leaving the choice of what to focus on just before the exam to the student, when the teacher is probably in a better position to give advice.
and if your interested – How to predict exam questions

 

And finally my  Holiday books 
I have just started Sal Khans The one world school house – very easy to read summary of how Salman Khan got to become “Bill Gates favourite teacher” and the logic behind what the Khan Academy does.
Thinking fast and slow and The chimp paradox are still as yet to be read.

 

Now where’s that bill or is it cheque….20% tip!

Thank you for the music – listening to music when revising

Posted on by

May and June are the traditional months when students around the world lock themselves away to revise for their exams.

In China for example over 9 million students will be sitting the university entrance exams.

Last May (2012) Teenagers at Xiaogang school in Hubei province were pictured hooked up to bags of intravenous fluids hanging from the classroom ceiling to boost energy levels during the revision period. An extreme action by anyone’s standards, but perhaps an example of how much pressure students feel this time of year.

As I mentioned in last month’s blog my daughter is currently caught up in this May/June exam frenzy. So once again I found myself looking to her for inspiration. What was she doing, how did she revise? This is not because she is a perfect example of a revision student, in many ways she is not, but I do think she is typical of many.

What does Beth do?

  • Makes notes from her notes – This is a standard exam skill, reducing content down into measurable and personal chunks. She does use mind maps (possibly my influence) but not exclusively.
  • Prepares as if she has to teach someone else – this I find interesting and has certainly not come from me. She writes on a white board the key points as if she was going to teach that subject. I like this idea, as many teachers and lecturers will tell you nothing focuses the mind nor motivates you more than having to teach it to others.
  • Practices past exam questions…of course!
  • Studies while listening to music – now this is the one that intrigued me and as a result I have devoted the rest of the blog to answering the question …..

Is it a good idea to revise whilst listening to music?

As ever the science needs much interpretation.

It’s a bad idea

Researchers from the University of Wales, tasked 25 students with memorising lists of consonants. Some were shown the letters while sitting in silence, others while listening to music by their favourite bands or by groups they had a strong aversion to. The conclusion was that listening to music, hampered their recall.

So it’s good then

Scientists at Stanford University, in California, believe there is a molecular basis for music known as the “Mozart Effect“. It was discovered that rats, like humans, perform better on learning and memory tests after listening to a specific Mozart Sonata in D.

But then there is the evidence that suggests that switching attention when trying to learn as might be the case with listening to music slows down the cognitive process.

Yet you cannot ignore the research that clearly shows music has the ability to alter your brain and induce relaxation which in turn helps create an ideal state for learning.

Watch what happens to your brain when you listen to music.

Hopefully you get the idea.

Conclusions

  • Listening to music puts you into a more relaxed frame of mind and that is of course good for learning. So listening to music before or after revising can help.
  • If you do want to listen to music, avoid music that requires you to shift your attention. This would suggest you should not listen to  music with lyrics  as it can mean you need to think about what is being said nor should you listen to something new that you may not have heard before. This is one of the reasons classical, in particular baroque music is the preferred choice of many students. Also don’t play the music too loud, keep it as background noise.
  • If there are specific facts that you simply need to know, then avoid listening to music completely, give it your full attention. But you can’t concentrate at this level all day, only for short periods.
  • On the whole be consistent don’t keep changing the type of music, you need familiarity.

Music to help you study

The internet has many websites that offer relaxing and helpful music, here are a few that might help.

There’s no such thing as a stupid question – Learning by questions

Posted on by

Ask questionsThe Greek philosopher Socrates was born 469 B.C in Athens, and died in 399 B.C. He is considered by many to be one of the world’s greatest thinkers. He is known for the Socratic Method and the pursuit of knowledge. The Socratic Method involves asking a series of questions until a contradiction emerges invalidating the initial assumption.

Socratic questioning does not seek to find THE answer, there are often many answers. The primary goal is to explore the contours of often difficult issues and to teach critical thinking skills. This method encourages you to go beyond the simple memorising of facts, enabling you to develop a higher level of understanding.

Socratic questions

1. Questions for clarification – Why do you say that?
2. Questions that probe assumptions – What could we assume instead?
3. Questions that probe reasons and evidence – What would be an example?
4. Questions about Viewpoints and Perspectives – What is another way to look at it?
5. Questions that probe implications and consequences – What are you implying?
6. Questions about the question – What was the point of this question?

If you can formulate a question – you have 60% of the answer
When you pass one exam the bad news is you are often faced with another. Exams at times can seem endless. But as you progress, what you are asked to do will change. When you first start studying a subject you will be asked relatively simple questions such as, what is the capital of FranceChildren are good at asking questions. To answer questions like this you need little more than a good memory. However when you get to the final level, examiners are more interested in understanding and application, not simply knowledge. What they really want you to do is think….

And so you may need to form an opinion, a view of your own. I am sure that you have many opinions now, but how informed are they, what facts support your view and how much have you thought around this view sufficient that you can deal with challenge.

This is where asking good questions can really help. When studying on your own, if you can formulate the right question you are more than half way to answering it yourself. Because to even get to this question, you will have had to think deeply about what you are trying to do, how it might work, what resources you might need, why has no one done this before etc. And only when you have thought this deeply will you be able to ask your question.

The next step is to post your question on the internet, someone will have the answer, the interesting thing is, by the time you come to do this, you may have the answer yourself!

Questions really are a great way of learning.

Judge a man by his questions rather than his answers.
Voltaire

The last word will go to Scott Adams
If there are no stupid questions, then what kind of questions do stupid people ask? Do they get smart just in time to ask questions?

Or should it!