In the training business, level one surveys are ubiquitous. Ever since Donald Kirkpatrick created the four levels of evaluation as a way to assess training programs, the level one survey, which is designed to measure students’ reaction to the training event is by far the most dominant. Many organizations invest heavily in level one surveys, including resources to design, implement, manage and report on the results. Individual or management performance reviews are based on the results of the level one survey.

You can think of the level one survey as a specific instance of a broader category of customer satisfaction surveys. Level one surveys in training are designed to measure students’ satisfaction with the training event. Level one surveys in, say, customer service, are designed to measure customers’ satisfaction with the service event, and so on.

Usually the surveys contain a number of different questions that deal with satisfaction, relevance, good use of time, meeting expectations, overall quality, effective delivery, recommendations and effectiveness. The number of questions on surveys can range broadly, as well as response rates.

In thinking about how to improve response rates, and looking at the number of types of questions contained in level one surveys, it’s important to consider how much information is really being conveyed by all these questions. Information in this sense refers to data that represent the inherent concepts being measured, is analyzed within a context that gives meaning, which leads to an increase in understanding, resulting in more insightful decisions.

To answer the question of how much information is really being conveyed, a statistical technique called “factor analysis” was used to analyze two satisfaction surveys.  Think of factor analysis as the statistical “analog” of affinity mapping which is used in many planning sessions. In affinity mapping, Post-it notes are used to describe ideas about some topic. These Post-it notes are then grouped by similarity into different categories and then each category is named. Similarly, factor analysis takes each variable as a Post-it note and groups like variables together (via analysis of correlations) into categories called “factors” and then names each factor.

Factor analyses were conducted on two separate satisfaction databases. One database contained a sample of about 120,000 level one training evaluations across nine variables measuring course perceptions (e.g., relevance, good use of time, met expectations, overall satisfaction, quality, etc). The bulk of the courses were one-hour to four-hour e-learning courses, although there were about 20,000 two-to four-day instructor-led training courses included. The second database contained about 3,000 observations of level one survey data from a customer services satisfaction survey across five variables measuring the quality of service delivery (e.g., overall satisfaction, professionalism, technical ability, etc).

Results of each factor analysis indicated that all variables were represented by a single factor.   This means that all the questions, in effect, measured the same thing (i.e., all questions were redundant).  The single factor could be called “satisfaction” or “likeability.” Students either liked the training or they didn’t; customers either liked the service event of they didn’t.

As a possible account for these results, the literature on episodic attitude formation was used to hypothesize that attitudes formed for short duration events are processed “holistically” based on the experiences, rather than constructed analytically based on a number of different perceptual or psychological dimensions. That is, the event in its entirety was judged. Given this holistic attitude, when asked to respond to the survey, especially with semantically similar questions, the pattern of responses is consistent throughout. It would be interesting to extend the analysis in the training context to factor analyze longer duration courses such as a semester, or seven-week accelerated trainings that many colleges offer. Perhaps with longer duration, attitude formation can be built attribute by attribute which would lead to multiple factors being derived.

A practical consequence to this small sample analysis, may imply that for short duration training, rather than asking many questions, one or two questions, such as overall satisfaction and recommendation would be sufficient. The resulting “shortness” of the survey may improve response rates as an added benefit. Further, many of the newer learning management systems provide “star ratings” for each course listed, so this rating system may be leveraged instead of the full-scaled level one evaluation. That way you can use your level one surveys in other ways to enhance understanding and better inform design and effectiveness decisions.

Bill Simcox is the program manager for Systems Documentation Inc.

Share