Organizations around the world spent $362.2 billion last year on corporate training, and every year since 2010, the global spend has increased. With so much of a company’s budget invested in training, it’s important for its leaders to make sure they’re seeing a return on that investment. Organizations see evidence of that return in improved employee productivity and performance back on the job. It’s also important to use assessments – before, during and after training – to ensure that employees are at least understanding and retaining what they learn. They can also be useful for reporting on compliance training to regulatory bodies.

Asking questions before, during or after training can improve knowledge retention as well as assess it, according to Brian McNamara, director of marketing at Questionmark. Asking a question and then providing corrective feedback “has a pretty dramatic learning benefit.”

Creating good assessment questions takes time, money and expertise. Fortunately, emerging technologies, such as artificial intelligence (AI), are making the process more efficient. For example, the Harbinger Group recently released a tool called Quillionz, which uses natural language processing to analyze training content and automatically generate questions for assessments. “A good multiple-choice question can take up to 15 minutes or more to design,” says Vikas Joshi, CEO and managing director of the Harbinger Group. Quillionz is an attempt to automate that process while maintaining the quality of an expert-created question.

Natural language processing (NLP) is a type of AI that uses algorithms to process, understand and manipulate language. Organizations are starting to use NLP in adaptive learning programs, chatbots and coaching. Creating assessments is another new application. “Once the questions are generated, as we know, AI is a technology which is still evolving, so it can only be better if we learn from user actions. So that’s where the machine learning comes into the picture,” says Joshi. Quillionz uses a machine learning model that observes user actions and uses that data to help create better questions in the future.

What Makes a Good Question?

Poonam Jaypuriya, vice president of e-learning at Harbinger Group, says that assessments should have two types of questions: retention-based questions and comprehension-based questions. Retention questions use fill-in-the-blank or “WH” (who, what, when, where, why) formats, while comprehension questions ask learners to reason, draw conclusions or select from multiple choices.

If your assessments are created by subject matter experts, McNamara recommends teaching them how to write good questions using coaching or an item writing workshop. Without good questions, you “detract from [the assessment’s] ability to distinguish between the people who know the information you’re testing against and those who don’t.”

Of course, that means that when a computer is creating questions, expert oversight is required to make sure the questions are good ones. Harbinger Group has a team of instructional design experts to review the questions the AI creates and “delivers an array of curated, handpicked questions,” according to the press release announcing Quillionz.

Getting Started with Technology

Joshi says it’s important for organizations to understand “the central role that language plays in learning and development.” If you’re interested in using tools like Quillionz in your assessments, make sure to “pay attention to not just the analytical technologies but also the language technologies around AI.”

Other important considerations include responsiveness and security, says McNamara. Make sure your online assessments can be accessed on any device and using any browser, especially if you have a large deskless population. And while technology can make assessments less secure in some ways, online proctoring technologies and other innovations can improve security and reduce the risks of both cheating and data hacks.

“Ultimately, you need to be able to trust the platform that it’s being delivered from,” says McNamara. That includes being sure that the assessments and the data they collect are secure. It also means trusting them with your learners – making sure that the technologies they use are enhancing assessment and learning, not being used just for show. Finally, says Jeremy Auger, chief strategy officer of D2L, if you’re using AI in your assessments, it means ensuring that you don’t include “historical bias” in the data that you use for the algorithms. (An example from hiring: “If you traditionally hired males in technical roles, does the data driving the AI continue to filter women from the process?”)

“There’s an ever-growing need for questions [that make] it easier for people to learn, either on their own or through the help of great learning portals,” says Joshi. Regardless of what tools or platforms you use, making sure you’re efficiently – but effectively – enhancing the learning experience should be your primary goal.