AI in Corporate Training: How to Tell a Real Claim From a Demo
Most AI training pitches demo the part that was never slow. The four questions I ask before believing an AI training claim, and what to build capability in now.
Every AI training demo runs the same script. Someone drops a policy document into a tool, and out comes a module — objectives, scenario, quiz, narration — in the time it takes to refill a coffee. The room is impressed. Nobody asks the question that matters: was producing the module ever the slow part? Having built learning systems since 2018, including a long partnership with a university dental program, I can tell you it almost never is. The slow parts are deciding what to build, getting an expert to confirm it is correct, and learning afterward whether it changed anything.
Generation demos well because it is visual and instant. Analysis and evaluation demo badly, because both take weeks and produce a document rather than a screen. So the market optimizes for what looks good in a thirty-minute call. Before you watch anything generate, write down where your last project actually lost time, then judge the tool against that list.
Four questions I ask before I believe an AI training claim
- What is the model actually doing — generating, retrieving, scoring, or routing? "AI-powered" flattens four capabilities with four failure modes. Generation invents; retrieval surfaces what you already approved; scoring judges a person; routing decides who sees what. A vendor who cannot answer feature by feature has not thought about it.
- When the output is wrong, is human review forced or merely permitted? Every tool will tell you a human can review. Ask whether anything stops an unreviewed module from reaching a learner. A review someone can skip on a deadline gets skipped on a deadline.
- Whose content is it drawing on, and can I see the source of a given sentence? A system grounded in your own approved policies, with the source visible, is a different product from one improvising against general knowledge. In regulated content, that is the whole risk profile.
- What do I keep if I leave in two years? The course files, the records in a portable standard like SCORM or xAPI, and the configuration that makes it behave as it does. If those live only inside the vendor, you are renting your own training program.
What actually changes inside the design process
AI redistributes effort across ADDIE rather than removing it. Development — building, formatting, drafting a first knowledge check — genuinely compresses. But the weight lands on the two phases most teams already shortchanged. Analysis gets heavier, because producing a module in minutes makes it cheap to build the wrong thing at speed. Evaluation gets heavier, because more material now reaches learners than any one designer read closely.
Design changes character too. Instead of writing the content, you write a specification precise enough for a machine to execute: the objective, the audience, the constraints, the things that must never be said. That is closer to instructional design than to prompting.
Personalization is a data problem before it is an AI problem
Adaptive paths are the most oversold idea here, for an unglamorous reason. To route someone sensibly you need their role, their tenure, what they have already completed, and a signal about how they are doing in the job. Most organizations keep that scattered across an HRIS, an LMS, and a spreadsheet somebody maintains by hand. Until those are wired together, "adaptive" means re-sequencing slides inside a single module.
There is a design trap as well. Sweller's cognitive load theory is reason enough to distrust an interface that keeps branching, recommending and interrupting: every extra decision is attention not spent on what you are teaching. The well-evidenced part of personalization is feedback, and Hattie and Timperley's work is about telling a learner where they are, where they are going, and what to do next. A generated paragraph of encouragement is not that. Naming the specific gap and the specific next action might be.
What I would build capability in this year
I would not start with a platform decision. I would build four things into how the team works, because they hold value whichever vendor wins.
- Specification writing. An objective, an audience, a constraint set and a definition of done, stated clearly enough that a junior designer or a model could execute it.
- Adversarial review. Reading generated material for what is confidently wrong, not for typos. It is a distinct, trainable skill, and it is what keeps AI output from becoming a liability.
- Assessment design. Questions that still discriminate a week later, asked outside the module, without unlimited retakes.
- Owning the plumbing. The connections between your HR system, your LMS and your reporting, on infrastructure you control — we use self-managed n8n — so the workflow layer stays yours when the AI layer changes underneath it.
The teams getting real value from AI in training are not the ones with the most advanced tool. They knew their bottleneck before they went shopping, and they kept the judgment, the source of truth and the records on their own side of the line. That is what survives the next demo.
Frequently asked questions
How do I evaluate an AI training vendor if I am not technical?
You do not need to be. Ask the four questions in plain language and listen for specificity. Someone who can name which features generate, which retrieve from your material, what forces a review and what you can export is describing a real product. Adjectives in place of answers mean you are being sold a demo.
Should we wait until the tools settle down?
Waiting is its own decision with its own cost. Build the capabilities rather than the commitment: specification writing, adversarial review, assessment design and integrations you own do not depreciate when a tool is replaced.
Is this different for a small team with no L&D department?
The questions are the same, but the answers matter more. A small team has less slack to absorb a bad module reaching a learner, and less leverage to get its data back out later. Weight portability and forced review heavily, and wire the systems you already pay for first.