Training Effectiveness

A grey-haired woman wearing glasses is behind a service window. A cyan folder is between her hand and a magenta-sleeved hand, and a yellow folder sits on the counter.

To measure team AI training results, you need a task with criteria fixed before the course starts. An employee completes a comparable task, and a designated reviewer checks each answer against source facts. If something fails, you find the cause and pick a response. Below is the full walk-through, built on real public terms for team AI training.

Terms on the training page (October 4, 2026)

The page at majento.ai/ru/ai-training/ lists these conditions:

  • Team training starts at $2,600. That is a floor, not a final price.
  • Groups cap at 15 people. Larger headcounts get separate groups discussed.
  • In person is preferred. Online is possible.
  • The program adapts to the client's tasks and processes.
  • Final program, format, and cost are agreed before the start.
  • Which data may be used is agreed with the team.
  • AI champions are designated and implementation projects are defined.

The page does not promise to start implementation right away. It also does not say why in person is preferred. We do not add those facts here.

How do you test an AI skill with a real work task?

A good task is tied to a real work situation, not to repeating text from the course. The employee gets a task that differs from the training example but requires the same skill.

Below is a training example for an employee preparing for the AI champion role (the team member who helps colleagues use AI at work). The conditions come from the public page majento.ai/ru/ai-training/ (October 4, 2026). The answers were written by the author for illustration and were not run through a model.

Task (copyable text):

A team of 20 requested team training. Three requirements: online format, a final price, one group.

Conditions agreed before starting work:

  • Groups cap at 15. Larger numbers get separate groups discussed.
  • Team training starts at $2,600. That is the minimum.
  • In person preferred, online possible.
  • Program, format, and final cost agreed before the start.
  • Permissible data agreed with the team.
  • AI champions designated, implementation projects defined.

Write a response covering all three parts of the request (groups, price, format), accounting for every condition. Draft it for review, not as a final letter to the client.

Answer that passed review:

  1. On groups. 20 people do not fit in one group because the cap is 15. For larger numbers, separate groups are discussed. We agree on the program and format before the start.
  2. On price. The minimum for team training is $2,600. Final cost, program, and format are agreed before the start.
  3. On format. Online is possible. In person is preferred. Final format is agreed before the start.
  4. On data. Before starting, we clarify which data may be used in examples and exercises.
  5. On implementation. AI champions will be designated and implementation projects defined.

Answer with errors:

  1. On groups. We will fit 20 into one group, no problem.
  2. On price. Final price is $2,600, 10% discount for online.
  3. On format. We run it online, in person is not needed.
  4. On implementation. We start implementation right after the course, two-week timeline.
A brass two-pan balance on a white cabinet. The left pan holds a sheet with many red marks and strike-throughs, the right pan the same sheet with a single red mark. A hand steadies the top of the scale's post.

What counts as a passing answer and what gets returned?

Every claim in the answer is checked against a specific input fact. The reviewer checks factual accuracy only - not style or speed.

Acceptance criteria:

  • The numeric limit (15 people) is not violated.
  • Price is named as a minimum, not a final figure.
  • Format is stated as the source states it: in person preferred, online possible.
  • No implementation promises the source does not contain.
  • No discounts, exceptions, or timelines the source does not contain.
  • Mention of program and data agreement is present.
  • Mention of AI champions and implementation projects is present.

Answer that passed review:

Claim Status Source fact
20 people do not fit in one group Accepted Groups cap at 15
Separate groups discussed for larger numbers Accepted Separate groups discussed for larger numbers
Program and format agreed before the start Accepted Program and format agreed before the start
Minimum $2,600 Accepted Training from $2,600 - that is the minimum
Final cost agreed before the start Accepted Final cost agreed before the start
Online possible Accepted Online possible
In person preferred Accepted In person preferred
Final format agreed before the start Accepted Format agreed before the start
Clarify permissible data Accepted Permissible data agreed with the team
AI champions and implementation projects Accepted AI champions designated, implementation projects defined

Answer with errors:

Claim Status Violated condition
20 people in one group Returned Cap is 15
Final price $2,600 Returned $2,600 is the minimum, not the final price
10% discount for online Returned No discount in the source
In person not needed Returned In person is preferred
Implementation immediately, two weeks Returned No commitment or timeline in the source

Review is based on source facts from the public page. The returned claims contain numbers and commitments that do not exist in the source. That is what makes them wrong.

Memo to the manager after review. A training answer covering three parts of the request was reviewed. Ten claims in the correct version held up against source facts. Five claims in the version with errors were returned, each with the specific violated condition named. What was tested: the ability to hold source constraints while drafting a response - not the ability to write beautifully or quickly.

This review did not observe a real employee in a real situation. It also did not measure independence from the trainer, time spent, use of the skill on a different task, or completion of the full course. So it cannot show that the course paid off or that the team became more effective.

Decision on this sample: the task and criteria are ready to use in a real review with a real employee. Evaluating actual training requires the person to complete the task themselves, with observations logged against the same criteria. Without that step, this memo stays instructional, not final.

Limits of this example: the walk-through shows the review method works on a composed text. It does not show how the method behaves on a live employee who is tired, rushed, or never read the source fully.

Why do employees fail the check and what comes next?

One error does not yet tell you the cause. Diagnose before you choose an action.

Case 1. Employee wrote "20 people in one group." A numeric constraint from the source was violated. Diagnosis: did the employee see the 15-person limit, and was that limit clear enough in the source? If the limit is visible but was skipped - give practice checking numeric facts in sources, not a full retake of the course. If the limit was buried in a long paragraph - fix the source wording, not the employee.

Case 2. Employee wrote "10% discount." A number was added that does not exist in the source. One defect does not establish the cause. Possible causes: the employee did not see the source; or saw it but did not understand that "from $2,600" means a minimum; or the task did not ask for an exact price, so the employee made one up to sound complete. If the cause is misunderstanding "from" - walk through one example together. If the task did not require an exact price - change the task and add a section on agreeing the cost. If the employee is used to adding extras for the client - give practice distinguishing what is in the text from what is invented.

Case 3. The needed condition was not in the task. Suppose the source contained no condition on format. The employee could not comply with something that was not there. Action: prepare a source with a complete list of conditions, do not change the criterion after the fact, and do not blame the employee.

Case 4. The task does not test the needed skill. The "respond to three parts" task shows the ability to hold constraints. But if the team's real job is to negotiate program scope against client business tasks - that part needs to be added to the task. A formally correct answer may not show the needed skill.

In none of these cases is retraining assigned by default. Diagnosis comes first. The action follows its result.

A brass hourglass with a paper price tag tied to it by red-and-white string. A hand lifts the tag, revealing check marks in boxes and unreadable lines on its back.

What to agree before the course to measure team AI training results

Evaluation only works when criteria are written down before the start, not invented after the fact.

There are four levels of evidence:

  • Attendance. The person was in the session. That is a fact of presence, not a fact of understanding.
  • Specific completion - the employee performed the assigned exercise against pre-set criteria. Shows the skill is accessible under controlled conditions but does not mean the employee will apply it independently.
  • Transfer to a work task - the employee solves a comparable but different task. Shows the skill is not locked to one example but does not prove a change in business results.
  • Business effect. A change in payment, processing time, or rework volume. Requires separate measurement and does not follow automatically from the three levels above.

Which level is needed depends on the role. A support team may stop at transfer. A development team may go straight to business effect by measuring debug time. Requiring the same level from everyone does not make sense.

Before the start you also fix: who the reviewer is (a role - for example, the department head who requested the training; the name is not invented, only the role is recorded), what task is given, which criteria determine a passing answer, and how many people must complete the check before the result can represent the team.

AI is allowed during completion. If the trainer gave a hint, that is recorded separately. Until the employee manages without a hint, independence is not confirmed. But a trainer hint does not mean the training was useless.

What gets recorded. The employee's draft, time taken, what changed after review, which facts were missed. A failed attempt stays in the record - it is not replaced by the next attempt without a note. One completion shows the skill was available in that specific case. It does not prove the employee understood the entire course.

Full time to acceptance includes: preparation (reading the source, framing the task), completion (writing the answer), review (checking against criteria), and revision (fixing returned claims). A failed attempt counts inside that time and is not subtracted.

Absent and failed attempts are not removed retroactively. A before-and-after comparison alone does not prove the course caused the change: improvement may come from practice, changed tasks, or experience outside the course.

How do you count the cost and financial result of AI training?

The financial return on AI training cannot be taken from someone else's research and plugged into your own calculations.

What counts as additional spend. The course cost is an additional spend if it represents a real payment beyond what the organization would have paid without it. A course that fits the budget is not automatically extra spend. What matters is the actual payment and the option you chose not to take. Employee hours in sessions, the room, and equipment are resources the organization already owns. Hours in training can be converted at an internal rate for planning purposes - that is a monetary estimate of time cost, not ROI.

Freed capacity. If an employee completes a task faster or with fewer rework cycles, hours open up. The potential for more useful work is not equal to paid sales. Freed hours affect the financial result only when they are filled with work that generates payment beyond the current volume.

Direct savings. If rework drops and that means no payment for re-development, that is a specific mechanism of savings. Observing "there is less rework" without a mechanism stays a feeling, not evidence. ROI is unknown without your own before-and-after measurement. Plugging in someone else's numbers is not valid. No layoffs or savings are promised.

What does research say about measuring AI training results?

Two studies point to the same thing: measure the specific workflow and the specific task, not the fact of attending a course.

arXiv 2304.11771v2 (November 6, 2024) - 5,172 support workers - found that connecting an AI assistant gave an average 15% increase in resolved tickets per hour. Senior workers gained a small speed edge and a small quality drop. This is the effect of a workflow change from connecting a tool, not the effect of a training course.

A training example of a wrong phrasing: "According to research, after our course your team resolves 15% more tickets per hour." This sentence was composed for illustration. It transfers the effect of connecting a tool onto the effect of a course - that is incorrect. For an evaluation plan, the takeaway is: specific completion, transfer, and business effect are checked separately, not summed up from one figure.

The technological frontier of AI is the dividing line between tasks current models handle well and tasks they do not yet handle. The effect depends on the specific task and cannot be known in advance. HBS AI Institute (2023) - 758 consultants - found that tasks inside the frontier produced positive effects, and the result depended on the type of task. For an evaluation plan, this means: choose tasks the team actually performs, run a trial, and do not expect identical effects from different kinds of work.

The plan described in this article is not fully validated by these two studies. They point to one thing: measure the specific workflow.

Frequently asked questions

Can training count as successful if everyone attended and gave positive feedback?

Attendance and positive feedback show interest and satisfaction. They do not show the skill is available at work. A person can be satisfied with a course and still not use AI when solving real tasks. For a real check, you need a specific task with criteria and completion by the employee themselves, not by the trainer. One completion shows the skill was available in that specific case - that is already more than a course review.

What to do if an employee fails the fact-check?

Start by finding the cause. If the employee did not notice a constraint in the source - give practice checking numeric facts in sources. If they did not understand the structure of the source - walk through one example together. If the condition was buried in a long text - fix the source itself, not the employee. A retake or additional training is assigned only after the cause of the error is established. Diagnosis always comes first.

Is a universal task set needed for all teams?

No. Criteria are tied to what the specific role does. A support employee answers questions about training conditions - useful output requires holding source constraints. For a calculations team, useful output is different: correct arithmetic, the right formula choice, no rounding where it distorts results. For a development team: working code, test coverage, no new bugs in code that used to work. The universal principle is one: the criterion is tied to what the role actually does.

How do you measure business effect without invented numbers?

Before the course, define the mechanism - what exactly changes in payment or time if the skill transfers to work. Measure the indicator before the course. Run the course. Measure after. Compare. Without a defined mechanism, "we are faster now" stays a feeling, not a measurement. Correlation between the course date and a metric change does not prove the course caused the change: workload may have shifted, tools may have been updated, part of the team may have turned over. A before-and-after comparison without a group that skipped the course shows a difference, not a cause.

What is the AI technological frontier and why does it matter for evaluation?

The technological frontier is the dividing line between tasks current models handle well and tasks they do not yet handle. The effect depends on the specific task and cannot be known in advance. For an evaluation plan, this means: choose tasks the team actually performs, run a trial, and check the result on those exact tasks. Even on tasks that seem inside the frontier, the outcome is not guaranteed until a trial run happens.

How do you choose a review task that differs from the training example?

The task must require the same skill but put the employee in a different situation. For example, if the course used a request from a team of 20, the review gives a request from a team of 12 with different requirements. The acceptance criteria stay the same: numeric limits, price as minimum, format, data, implementation. This tests the transfer of skill, not the ability to remember one specific example.

What goes in the manager memo after review?

The memo contains: what was reviewed and under what conditions, how many claims passed review and how many were returned with the specific reason, which criterion was being tested, what was not observed during the review (independence, transfer to a different task, completion of the full course), and what action is recommended next. A memo written on a composed text without observing a real employee stays instructional, not final.

Should the review happen right after the course?

The timing of the review is fixed in advance along with the other criteria. Delaying the review is not forbidden, but the longer the gap between the end of the course and the review, the harder it is to separate the course effect from other factors - practice, changed tasks, experience outside the course. If the timing was not fixed before the start, the result is harder to interpret.

Conclusion

To measure team AI training results, agree on criteria in advance, give a comparable task, and check the answer against source facts. Course attendance and positive feedback do not show that the skill has transferred to work. Financial results require separate before-and-after measurement, not borrowed figures. Diagnosing the cause of an error matters more than assigning retraining immediately.

If you want to build this evaluation system for your team, start with corporate AI training. Questions: hello@majento.ai | Telegram @shimaoz

Sources

  1. arxiv.org publication
  2. aiinstitute.hbs.edu publication

View as Markdown