The single biggest challenge facing learning leaders today is evaluating the effectiveness of their learning interventions. Currently, data analytics is all the rage. There is a vast array of analytical models, tools and data science expertise available, all of which are working to transform businesses across the globe.

So, how do we leverage the latter to enable the former and give learning and development (L&D) organizations accurate tools to measure the effectiveness of their learning programs and use that data for continuous improvement and transformation?

The Challenges Faced by CLOs and Their Teams

L&D professionals understand that learning is crucial to business performance. However, proving the value of L&D to other parts of the organization, including business leaders and decision makers, is an uphill task. Why do chief learning officers and their teams struggle in this area? The answer lies in the practical challenges they face in measuring the effectiveness of learning programs and, hence, the challenge in showing their impact on business goals and performance.

There are frameworks that address measuring learning effectiveness, the most notable being the Kirkpatrick Model:

Level 1: Reaction
This level refers to the reaction of learners, typically captured through feedback forms or “smile sheets”; the reaction phase helps L&D professionals understand to what degree participants find the learning intervention engaging and relevant to their jobs.

Level 2: Learning
This level refers to the degree to which learners acquire the intended knowledge, skills and attitudes addressed in the learning intervention, and it is typically measured through assessment scores.

Level 3: Behavior
This level refers to the degree to which learners apply their learning from the learning intervention to their jobs; supervisor, mentor and 360-degree feedback are some ways to capture this information.

Level 4: Results
This level refers to the degree to which targeted outcomes occur as a result of the learning intervention and its impact on key business metrics.

However, many organizations following these frameworks lack the tools needed to put the framework to use. At each level of the Kirkpatrick Model, the first step toward measuring effectiveness is data collection which, for many organizations, is a difficult task to carry out beyond Level 2.

According to the “State of the ROI of Learning” report by Udemy, most companies still rely on Level 1 metrics and were primarily measuring training satisfaction and completion rates. By relying only on Level 1 data, organizations fail to measure the L&D’s impact on employee behavior, key performance indicators (KPIs) and critical business metrics.

Even with the wide availability of data, how does one go about analyzing the mass of information to generate valuable insights that enable better decision making? This remains the bigger challenge.

This is where data mining and analytics come into the picture. The key objective is to start a thought process on how to address the challenge of measuring learning effectiveness by leveraging cutting-edge, new-age technology. How can organizations leverage analytics to evaluate the effectiveness of their L&D interventions and drive continuous improvement?

Data Mining and Analytics in L&D

Data mining is the process of finding anomalies, patterns and correlations within large data sets to predict outcomes. Big data analytics is the process of collecting, organizing and analyzing large sets of data (called big data) to discover patterns and other useful information.

So, how can L&D professionals leverage data mining and analytics? Figure 1 is a depiction of what analytics tools can be leveraged and at what level of the Kirkpatrick Model.

Let us delve into a little more detail, with the four Kirkpatrick’s levels as the reference points.

Reaction

Organizations typically collect learner feedback through either physical or online survey forms. In the case of e-learning hosted on learning management systems (LMSs), feedback forms are often built into the e-learning course.

The net result is a host of data (from questions that ask learners to score parameters such as the relevance of course content, ease of understanding of concepts, interactivity or engagement, etc., on a five or 10-point scale) and/or a large amount of unstructured information through comments on various parameters in the questionnaire or survey forms.

To analyze this kind of unstructured data, there are two analytics tools that, when used in tandem, are extremely useful: text mining and sentiment analysis. Imagine a Python sentiment analyzer that takes input from the various feedback sources, the algorithm generates sentiments from the required or specified surveys, throws up key themes and generates output that is continuously appended to the previous analysis for the same training program. Then, it generates dashboards with meaningful, insightful data and shows the progression of trends over time.

Consider the following example: Let’s say that a large global organization conducts a new manager induction program for people who have been newly promoted to managerial or supervisory roles from individual contributor roles. Feedback is gathered and run through the Python sentiment analyzer every time the training is conducted. From each session, the training team obtains instant insights on what worked well, what needs improvement, etc. Insights can be generated in numerous ways, such as geographically and evaluation parameter-wise. So, for example, course content might be working well across geographies. Logistics might be working well in one geography but might be a pain in another. Learner engagement might be a pain across the board.

Text mining reads each sentence in verbatim from the feedback forms and then breaks, chunks and organizes the information. Then, it throws up key themes in visual formats (such as highlighting pain areas in red and successes in green).

Now this is something L&D teams can work with! They now know what specific aspects are working and which are not, which geography/group/department is performing well and what can others imbibe from them, and a host of other meaningful insights. This helps L&D teams create continuous improvements over time.

Learning

LMSs today can generate a host of useful insights from learner-generated data. They can also generate individual and group proficiency dashboards by processing learners’ pre- and post-assessment scores or assigned target score levels and achieved score levels. Reports can also be generated to compare the scores of managers, departments, specific groups and/or locations, in addition to showcasing trending scores and the progression of scores over time. Overall, artificial intelligence (AI) powered LMSs have the ability to scan the host of data and visually categorize scores, depict trends and flag anomalies.

This empowers L&D teams much more than traditional LMSs that provide insights solely related to the amount of time learners spent on the course, individual learner scores and the most popular courses. Thus, L&D teams can accurately quantify the learning that has taken place in the organization as a result of L&D interventions.

Behavior and Result

This is where AI and predictive analytics come in. Data on behavior and results over time are typically derived from outside the LMS, from systems such as HR management systems (HRMS), performance evaluations, customer surveys and/or 360 surveys. Today, even external data sources can be plugged into the learning system to provide a more accurate employee profile. Systems such as Adobe Captivate Prime have features that enable the automatic importing of user details from HRMS or other applications, such as Salesforce Dot Com (SFDC), into the system. Thus, the system can recommend learning paths based on a combination of assessment scores and performance data. This is the essence of predictive learning.

Predictive analytics is embedded in several learning platforms today. Platforms like EdCast and several others apply AI and machine learning algorithms in order to provide corporate learners with the online training resources they require. The resources are pulled from the LMS as well as external sources, such as YouTube videos, TED talks and learning libraries on the web. As a result, every member of your team can pursue personalized online training paths to bridge gaps and improve workplace performance. Learners are given recommendations based on their performance according to key metrics, assessment scores, what courses they are viewing already, what courses others in similar roles are viewing, similar learning needs and/or similar learning interests.

The result is wholesome learning integrated with performance and behavioral aspects. Training paths can be re-adjusted based on behavioral and performance changes, and the data is available to managers to inform evaluation feedback.

How to Kickstart and Sustain Leveraging Analytics for Learning

L&D professionals should not shy away from asking for support. They will need to work closely with big data and analytics experts to see where they are in terms of the quantity and quality of learning data, what improvements need to be made and the time and effort that will entail. It is the data that is key to showing results. L&D professionals will also need to work with technology experts to understand what exists in the organization, to evaluate what features their LMS supports (assuming there is an LMS in place) and whether they are optimally utilizing the LMS; they should also evaluate whether a change in LMS implementation is required. Adopting an LMS that integrates with the organization’s HRMS and curates and aggregates content from external sources — while also employing AI and machine learning for personalized learning — is the way forward.

Ideally, this process should be followed up with an implementation roadmap and associated budgets, with relevant stakeholders enlisted for approvals. This would aid in ensuring the L&D team is committed to driving business results by understanding the value of training through effective training evaluation practices.

Benefits of Leveraging Data Mining and Analytics in L&D

Having the right data and accurate insight into learning effectiveness will help drive continuous improvement and performance enhancement across the business. Through these tangible contributions to the business, L&D can establish itself as not just a cost center — but as a business partner.

Insights on skill levels, the alignment of employee skills with business needs, and the impact of learning on key organizational metrics can enable more informed decision-making. As a result, L&D can better determine what courses need to be designed and delivered, and can make better decisions related to hiring, staffing and competency development as a result.

By leveraging data mining and analytics in L&D, business stakeholders can see how the learning organization is addressing and impacting key business imperatives, including operational efficiency, employee performance and, ultimately, business results.