The problem of training transfer and its most recent synonym, “scrap learning,” have been the nemesis of learning professionals for decades, yet very little has changed beyond identifying them as a problem. Luckily, there is a solution. I’ve coined it Learning Effectiveness by Design. It’s a systematic method for improving the effectiveness of key learning solutions by bringing together instructional design skills, informal learning techniques, change management tactics, and evaluation and assessment strategies. This approach requires reframing the traditional role of the learning professional in order to harness the benefits of each of those elements.
The Learning Effectiveness by Design approach begins with one key question: Does the learning design achieve the desired performance outcomes? If you can’t answer this question for a key learning solution, you will be able to after working through this thought process.
When to Use Learning Effectiveness by Design
This approach is not applicable to learning solutions geared at achieving general awareness about a topic. Rather, it is designed for a key learning solution that is intended to achieve a behavior modification and meets one of these criteria:
- High investment: Does it have a high dollar and/or time investment per learner?
- High impact: Does it impact most of the organization, a specific target audience or a business unit?
- High priority: Is it a key enabler of a broader organizational priority and/or change effort?
If any one of the following three key criteria is met, then it’s worth investing the time to use this approach, especially if you are in the process of developing or redesigning the solution.
The Learning Effectiveness by Design Mindset
A linear ADDIE mindset does not work for key programs that aim to change a behavior. This may be a tough pill for learning professionals to swallow, but the fact that the transfer problem has existed for decades makes this claim highly plausible.
Despite its utility, the ADDIE mindset results in a myopic approach to designing a learning solution. Learning professionals become bogged down in tasks like assessing training needs by interviewing learners who don’t know what they don’t know, crafting perfect learning objectives that conform to Bloom’s Taxonomy, battling with subject matter experts over content and writing elaborate facilitator guides that aren’t used. All of these tasks are important; however, focusing on ADDIE consumes most of learning professionals’ time. As a result, they don’t spend enough time assessing the gaps in their learning culture. Addressing these gaps is crucial to achieving the intended behavior modification.
A Learning Effectiveness by Design mindset moves a training professional’s thinking away from learning objectives and shifts it to performance outcomes. Learning effectiveness means that a key learning solution has proven to (a) modify the learner’s behavior (b) to such an extent that his or her performance has improved, (c) resulting in a measurable and positive business impact.
When a learning professional adopts a Learning Effectiveness by Design mindset, he or she will develop a habit of asking this question: “Which learning design(s) or intervention(s) at an individual, team, organization and client level would help achieve the key performance indicators?” A learning design that answers this question will better match the organization’s learning culture than one that achieves the stated learning objectives.
Key Behaviors
Here are four key behaviors that drive the Learning Effectiveness by Design mindset, along with steps to help you along the way.
Start with the end in mind. Identify the key performance indicators (KPIs) of the learning solution at an individual, team, organization and client level. Before you begin designing the learning solution, meet with the sponsor, and present five or six key performance indicators that the learning solution could directly and indirectly impact. Consider business impact KPIs, such as increased retention of key talent, early advancement of key talent, readiness of key talent to take on a role in the leadership succession pipeline, increased revenue, increased adoption rate of a strategic tool/process or cost savings.
Then consult with the learning solution sponsor to narrow the metrics to three KPIs. Limiting the number of KPIs makes the learning solution more focused. If you have identified eight or more KPIs, it’s likely that the learning solution is part of a multi-year, organization-wide initiative, and you’ll need a multi-year timeline that focuses on three or four KPIs per year. Furthermore, use metrics that matter to the sponsor, and frame them using terminology that resonates with senior management, not learning professionals.
Use a scientific approach. Harness a scientific research methodology to design an effective learning intervention and test if your interventions will achieve the KPIs. Accept that the post-course evaluation survey is the weakest evidence you can use to assess learning effectiveness. Put more effort into obtaining metrics that provide evidence of learning transfer and business impact.
Adopt a research mindset by designing a customized learning intervention that will achieve the desired KPIs. Proactively estimate the efficacy of each key design element of the learning solution in achieving the desired KPIs. Ask an independent learning or evaluation colleague to review the design of the learning solution. Invite him or her to constructively challenge the ability of the design to achieve the desired behavior modification.
Separate the role of design from the role of measurement. This separation will achieve the independence needed to ensure that the learning solution is effective. Too often, learning professionals are their own judge and jury. This mixed role contributes to scrap learning.
Collect data from multiple sources to ensure valid and reliable evidence of the solution’s business impact. Combine data sources by relying not only on self-reported information from learners but also on evidence from coaches, on-the-job supervisors, peers, direct reports and/or existing business data. Use both qualitative and quantitative data collection methods. This combination will provide a sound audit trail to prove that the learning intervention, not an extraneous factor, achieved the desired KPIs.
Devise a sustainability plan up front. Leverage change management and organizational development approaches to design and implement a plan that will ensure that learners will be motivated, supported, rewarded and recognized for modifying their behavior.
Identify any barriers for training transfer. How does your organization impede supervisor encouragement, feedback and coaching, or opportunities to use new information and skills? Utilize organizational development and change management tactics to break down these barriers. Proactively consult with colleagues experienced in these areas to identify interventions that drive transfer of training after the formal learning intervention.
Link the KPIs of the learning solution to the performance evaluation of on-the job supervisors. This alignment will help motivate supervisors to coach learners so they can demonstrate the desired behaviors on the job.
Report on the value delivered. Identify the program sponsor upfront, and agree on desired KPIs as well as a timeline to report the costs and benefits of the learning solution.
Communicate the value of the learning intervention from a cost-benefit perspective. For instance, provide evidence to demonstrate that you made the decision to use an external vendor for the formal training element based on a cost-benefit analysis. Discuss the ways in which this decision allowed sufficient time for designing and implementing a sustainability plan with on-the-job supervisors in order to ensure that the desired KPIs were achieved.
Use meaningful dashboards to present the progress on each KPI with a frequency (e.g., monthly or quarterly) upon which you have agreed with the program sponsor. These dashboards will ensure the continued support of senior management after the training event. Investigate data analytics tools you might use to analyze key information, spot trends and present KPIs in an effective and timely manner.
Leading and Lagging Indicators
The Learning Effectiveness by Design approach utilizes leading and lagging indicators to actively manage the work environment variables that influence the desired behavior modification.
Leading indicators are metrics that assesses whether a learning solution is on track, somewhat on track or not on track to achieving the desired behavior modification. If leading indicators are not favorable, you can take corrective action to proactively address gaps and reduce scrap learning.
For instance, assume that you have identified active coach involvement in securing stretch assignments (a leading indicator) as a critical element of a leadership development program. If only eight out of 15 coaches attend the briefing call to discuss their roles and responsibilities, the learning intervention is certainly not on track to achieve the desired KPIs. You could proactively react to this red flag by briefing the remaining coaches using a different approach, such as a 1:1 briefing. This process may seem time consuming, but without it, it’s very likely that participants will not obtain the right level of coach involvement needed to ensure their success after the program.
Use green (on track), yellow (somewhat on track) and red (not on track) on your dashboard to track leading indicators after a learning solution has been successfully implemented.
Lagging indicators represent the extent to which the end goal of a learning solution has been achieved. Unlike leading indicators, these metrics cannot be changed. For example, lagging indicators could include the retention rates of participants, early career advancement, and improved 360 assessment results one or two years after implementing the leadership development program.
It is often said that numbers do not lie. This statement is a compelling rationale for why metrics are at the heart of the Learning Effectiveness by Design approach. It guides the decisions learning professionals make before, during and after the formal training event and ensures that we develop and implement holistic learning solutions.
As a next step, select one learning solution that meets the criteria for this approach. Identify three KPIs and share them with a few colleagues. Develop a perspective on how you will design the learning solution to achieve the desired KPIs. Continue to experiment with the remaining behaviors and steps presented in this article until the Learning Effectiveness by Design approach becomes a habit.