Congratulations! You’ve been promoted in the learning and development (L&D) department because of your experience with remote learning. Sixty percent of your company’s employees now work from home, and your first major assignment is to develop a robust and effective remote learning strategy to replace in-person instructor-led training.
The senior management team is skeptical about the effectiveness of remote learning and whether the impact will be sufficient to justify the cost. Where do you start? How can you make sure that you are measuring something meaningful? And will you need any expert help? This article answers those questions to help you kick off your remote learning program successfully.
Know Why You Are Measuring Impact
Why are you measuring impact? It may seem like a silly question — of course, knowing the impact of learning is important. But there are several specific reasons an L&D team would need to measure impact, including compliance issues, performance feedback, evaluating the effectiveness of change management and demonstrating personal growth.
Ultimately, to evaluate any of these types of impact, measuring return on investment (ROI) is necessary — especially when it comes to replacing an existing L&D program with remote learning.
Begin by defining your goals:
- Are you comparing the effectiveness of remote learning to an existing on-site version of the same content? Doing so can be difficult if you have no prior measurements for on-site learning or are unable to randomize your sampling.
- Do you want to know if remote learning is effective for your employees?
- Are you comparing one modality of remote learning to another (e.g., the ROI of podcasts versus video)?
- Are there specific performance enhancements you wish to measure? In other words, do you want to find out if taking a particular remote course increase job performance?
- Are you trying to affect performance immediately (e.g., preventing dangerous or costly errors) or over a longer period of time (e.g., increasing sales and improving customer service)?
- Is the L&D project part of an employee satisfaction program?
Not sure about the answers to these questions? Ask yourself what problem you are trying to solve with your new remote learning program. Define your goals explicitly so that you know what you need to measure to show results. Once you have a clear understanding of why you are measuring remote learning, you can define what success looks like.
Decide What to Measure and How
Include a plan for measurement when designing your learning solution. Doing so will ensure that your measurements reflect your goals and aren’t simply a reflection of whatever is easy to measure. For instance, knowledge checks within an eLearning course aren’t representative of sales performance.
There are four broad categories of metrics: reaction, engagement, learning and transfer.
The learner’s reaction, or satisfaction level, tells you how he or she feels about the learning experience. Organizations often measure reaction with surveys and interviews. While it may be difficult to use this information in calculations of ROI, learners’ satisfaction with remote learning may offer insights for future programs and help you determine the feasibility of continuing remote learning when it’s not as necessary. How did the learners feel about the instruction? Did they find value, relevance, clarity and ease of use?
Engagement is often measured by completion rates. For compliance training, for example, you may just need to know whether the learners completed each part of a course. You can measure engagement with clicks, reaction times, scrolling and knowledge checks that demonstrate a learner either paid attention to the content or already knew it. Engagement is a common measurement used in remote learning, because it’s easy to measure and gives some accountability to unobservable employees.
To measure learning, it’s important to establish specific learning objectives based on your goals and required competencies. Techniques such as pre- and post-training assessments look at individual improvement. Professional assessments can also compare the impact of remote learning to the impact of other forms of training or of no training at all.
Learning objectives generally fall into two categories: knowledge objectives (which require memory recall or the recognition of specific information) and procedural objectives (which require learners to perform a task correctly).
Transfer is, perhaps, the metric most relevant to ROI. It asks, “Can learners apply the knowledge and skills covered in remote learning to their job?” Job performance is the best measure of transfer and is evaluated with job assessments, observations, surveys and interviews of stakeholders.
While you can analyze learning transfer at the level of the individual learner, let’s return to your senior management team’s concerns about the ROI of remote learning. What is the impact of remote learning to the organization?
In addition to instructional measurements, you will need to consider business data and reports, observations, surveys, and interviews with stakeholders. How did remote learning affect operational costs, profits, and compliance costs and risks? Don’t forget to consider societal and community benefits, too.
Here are some practical tips:
- Find out which data sources you already have, such as safety reports, human resources (HR) records of complaints, turnover and information technology (IT) help desk calls.
- Meet with stakeholders to discuss the data you have and what else you need to know.
- Create a model or diagram for how you think remote learning impacts your goals and objectives.
- Seek expert advice on which modalities and learning strategies might be best for your performance goals.
Seek Expert Help
You may have heard the expression “Garbage in, garbage out” used to refer to the importance of collecting good data. Measuring the right things and knowing how to analyze the data can be difficult. Who do you call for help?
- Instructional designers with experience or advanced education in measurement.
- Measurement specialists and psychometricians with at least a master’s degree.
- Process improvement specialists who use effective measurement techniques.
Now, you’ve done it! You decided on a goal, created learning objectives that will impact ROI and developed a plan to measure the impact of your remote learning program.