How to Measure Training Effectiveness

After implementing an elearning course, we often feel tempted to simply move on to other projects. This approach will satisfy the compliance requirements, but if you want to make your training truly effective, then your work is not done yet. After the course is rolled out, you need to figure out how the training translates to everyday work and, later, business outcomes. This means understanding how much and how well the trainees learned, how they are applying this knowledge, and what additional knowledge gaps you should address.

We like to use Kirkpatrick’s Four-Level Training Evaluation Model to evaluate the training effectiveness as this model provides a good overview of training-related metrics and outcomes.

The Kirkpatrick Evaluation Model was created by Donald Kirkpatrick, Ph.D., to define the four levels of training evaluation and has become an important part of delivering the full cycle of eLearning services. The four levels of evaluation are:

1. The initial reaction of the learner and their thoughts about the training experience
2. The resulting change or increase in knowledge following the training experience
3. The learner’s behavioral change and application of the new skills on the job
4. The results or effects of the improved performance on the business

Kirkpatrick’s four levels are designed as a sequence of ways to evaluate training programs. As we proceed through each of the levels, the evaluation becomes more difficult and requires more time, but it also provides more relevant and useful insights. It’s also worth mentioning that Kirkpatrick’s model for evaluating training effectiveness works well for all training modalities (elearning, instructor-led training, self-paced learning, etc.). So let’s take an in-depth look at the four levels of evaluation.

Level 1 – Reaction

At this level, we evaluate how the learners feel about the training or learning experience. The typical questions revolve around the training experience, satisfaction, engagement, etc. We can use this feedback to evaluate the learners’ perceptions, potential future improvements to the course, and justification for the training expense. Note that at this point we are discussing only the learning experience itself, not the outcomes or business impact that it might result in.

Why we want to measure reaction (level 1):

• We will know how the learners feel immediately following the training
• It’s an easy way to have immediate feedback
• We can make sure the training experience wasn’t disappointing
• We will instantly see ways to improve the course or an elearning module

Questions you can ask to measure reaction (level 1):

• Did you like and enjoy the training?
• Did you consider the training relevant?
• Was it a good use of your time?
• Did you like the venue, the presentation style, timing, etc.?
• Did you feel comfortable participating?
• How hard or how easy was it to participate?
• Do you think you will be able to apply the knowledge you received?

Tools you can use to measure reaction (level 1):

• Feedback forms
• Post-training surveys
• Follow-up emails
• Online evaluation
• Questionnaires

Level 2 – Learning

Level 2 measures how the participants understand and retain the intended knowledge and skills as a result of the training. In contrast with level 1, here we focus more on the knowledge base and the learning content instead of the feelings and perceptions. It is important to understand what and how much the participants are learning and what content is being missed.

Why we want to measure learning (level 2):

• We will instantly know if the knowledge transfer took place
• We will understand which specific topics weren’t addressed properly
• The responses can give us ideas for follow-up training
• We will know which training content has the potential to result in changed behaviors

Questions you can ask to measure learning (level 2):

These should be content-relevant questions based on the material presented in the course.

Tools you can use to measure learning (level 2):

• Online quizzes
• Post-training knowledge checks
• Simulated activities
• Topic discussions
• Reflections
• Questionnaires

Level 3 – Behavior

At level 3, we measure how the learners’ behaviors change as a result of the training. In simple terms, whether the learners are applying the knowledge and skills they received in the training. It’s important to note that while this measurement can show us whether the learners have actually learned the material, the lack of changes in behavior does not necessarily mean that they did not retain the training content. There could be other reasons for the lack of behavioral change, such as workplace barriers, organizational stagnation, resistance to change, lack of motivation, etc.

Why we want to measure behavior (level 3):

• We will know if the training had any impact on the performance
• We will know if additional training support is needed
• We will identify the barriers to behavior change
• We will understand how/if the knowledge transfers into skills

 Questions you can ask to measure behavior (level 3):

• Did the learners apply the knowledge on the job?
• Were the relevant skills and knowledge used?
• Was there a noticeable and measurable change in the activity and performance following the training?
• Would the staff be able to transfer their learning to new team members?
• Are the trainees aware of the change (or the lack of change) in their own behavior?

Tools you can use to measure behavior (level 3):

• Observation on the job
• Review of the performance documentation
• Ongoing assessment
• Interview with the managers
• 360-degree feedback
• Review of key performance indicators

Level 4 – Results

Level 4 allows us to determine the tangible results of the training. These could be reduced production costs, improved quality, improved efficiency, happier customers, increased productivity, higher employee retention, increased sales. It can be difficult to compile many of these metrics, but doing so is the true way for the organization to determine the return on investment (ROI) of its training expenses. It can also be challenging to identify whether specific outcomes are truly the result of the training. Level 4 requires both pre- and post-training measurements of the training objective.

Why we want to measure results (level 4):

• We will know how the training affected the business
• We will quantify the changed behavior in terms of the improved performance
• We can calculate the training return on investment
• We will understand what training experiences result in better business outcomes
• We will be able to better plan future training programs

Questions you can ask to measure results (level 4):

• Was there a change in business results?
• Do we get fewer customer complaints?
• How long does the production cycle take following the training?
• Did we notice a change in staff turnover?
• Has the rate of defects changed?
• Are we now operating on different profit margins?

Tools you can use to measure results (level 4):

• Profit and loss statements
• Department reports
• Quantitative performance indicators
• Bank statements
• Ledgers
• Shareholder statements
• Other business metrics

Finally, when evaluating the training effectiveness and its impact on operations, it’s important to consider all 4 levels of the model. Doing this will ensure that you have the entire picture of how effective the training was, and how it can be improved. While Kirkpatrick’s model is not the only tool available to evaluate the training effectiveness, it does work well for many different organizations from large manufacturers to elearning development companies. And because it can be equally helpful with elearning, classroom training, webinars, and other training modes, organizations have embraced the model as a reliable tool for evaluating the training effectiveness.