One of the most difficult questions to answer for both the learner and teacher is, “how do you know you or they are learning”?

On the face of it, the answer might seem simple, you are learning if you think you are, if what has been said makes sense and that you are happy with the way it’s being taught. But what does “make sense” actually mean? Presumably the process was logical and you understand it, you might even have found it interesting. However just to make sure you understood it, maybe we should ask some questions in the form of a test, and if you pass the test then that must mean you have learned it? But passing a test doesn’t mean you could apply it in a practical situation, isn’t that the best way to decide if it has been learned?

Each of these questions is doing something different, they are evaluating learning from a different perspective and if that seems the right approach you will probably very much like the Kirkpatrick model.  

The Kirkpatrick Model
In the 1950’s Donald Kirkpatrick developed the model that carries his name when he used it as the basis for his Ph.D. dissertation entitled, “Evaluating Human Relations Programs for Industrial Foremen and Supervisors.” But don’t let that put you off, the main point is, it’s a practical model designed to evaluate training not necessarily learning in a wider context or education. However as discussed in a previous blog, “Training V Education”, may not always be that far apart.

The model has four levels.

Level 1: Reaction – this measures the degree to which participants find the training useful, engaging, and relevant. Its focus is on the learner’s perception and as such is an opinion, which leads to the criticism that people may be happy but have learned nothing or vice versa. It is commonly evaluated using so called “happy sheets”, offering a range of smiley faces for the participant to indicate which one they think best captures how they feel, but they can be more sophisticated, with detailed questionnaires pre and post learning. This is the most popular method of measuring learning, with some suggesting around 80% of organisations use them.

Level 2: Learning – this is about the degree to which learners acquire the intended knowledge and skills and is most often evaluated using exams or tests. Critics will argue these only measure short term memory and not the longer-term deep learning that is required. Also, in practice the actual assessments are not always well written, partly because the expertise is not available to question the reliability and validity of the test.  

Level 3: Behaviour – this relates directly to how much the learners can apply in practice what they have learned. In the workplace this would be the application of what was learned reflected in improvements as to how the individual does their job. But it can also be assessed using assignments, case studies and real-world projects.

Level 4: Results – the last level is a measurement of the impact the training has had on the learning outcomes. Has the training achieved what it was originally designed to do. This is the least measured largely because of the difficulty and costs involved. Which is unfortunate because in many ways its the most important measure of success.

Kirkpatrick is far from perfect but the biggest criticism is probably in its application, with most organisations stopping at level 2 because 3 and 4 are too difficult. There are other models of assessment that some consider to be superior for example, the Philips model (ROI Methodology), the Kaufman model and Rob Brinkerhoff’s success case methodology, but obviously they are have critics as well.

Why does this matter?
Learning evaluation is clearly hugely valuable for educators, given the amount of time, energy and money that goes into training, it’s essential you know how effective it is. In addition, the Kirkpatrick model serves another purpose by helping with course design, just spin it around and ask the following questions. Starting with level 4, what do you want the training to achieve, what is the learning outcome? Then moving to level 3 ask, what behaviour will you see if learners are doing what is needed? Level 2, what knowledge and skills do learners need to do this? And lastly level 1, how do I want learners to feel, to help make the experience as effective as possible.

But it also has some value for learners, enjoying a course is important because it helps with concentration and to a certain extent motivation but the evidence shows, learners are not very good judges as to the effectiveness of learning. That’s not to say if something doesn’t’ make sense you shouldn’t challenge, just that sometimes you might find a particular exercise difficult, begin to question your level of intelligence and worry you’re not learning, when in fact the difficulty is a necessary prerequisite to embed the learning in the first place.

Lastly – What the Kirkpatrick model does is shift the perspective from measuring learning in terms of an emotion and the ability to answer a question, towards creating new improved behaviours and ultimately getting the results you want.

Footnote – the famous quote “Beam me up Scotty” is of course Captain or latterly Admiral Kirk speaking to Scotty his Chief Engineer. But Montgomery Scott to give him his full fictional name may have been a wizard with the Enterprise but unfortunately, he didn’t come up with a framework for evaluating learning.