AI Agent Skills Management
A process skill is a working instruction whose output goes to someone other than the author: a manager, a client, an auditor, an accounting system. When several people use it, one version splits into copies, each edited for personal preference, and fixing the shared skill as a whole becomes impossible. Research shows that AI is used widely but payoff is seen by few, and a lot of time goes to rework. One probable cause, which this article examines, is a team working from different versions of the same instruction. The fix involves four mechanisms: one owner for the skill, one version with a change history, tests before each distribution, and a weekly review of error flags. Below is how to set up each one for a skill in your own department.
Almost everyone uses the assistant, few see the payoff
In August 2026, 56.1% of US companies on the Ramp platform paid for at least one AI product: models, subscriptions, tokens (a token is a unit of text processed by a model), or API access. The number of active agents in the Microsoft 365 ecosystem grew 15 times in one year, 18 times in large companies; Microsoft does not disclose the baseline.
Payoff looks different. In the 29th PwC CEO survey (January 19, 2026, 4,454 CEOs from 95 countries), 56% of executives reported no revenue growth and no cost reduction from AI over 12 months. Both effects were recorded by 12%. McKinsey in "The state of AI in 2026" (August 25, 2026, 1,719 respondents, 97 countries) records that 37% of respondents attribute some effect on EBIT (earnings before interest and taxes) to AI. Nearly as many as a year earlier. The share who attribute 5% of EBIT or more and call the effect significant (McKinsey calls them high performers) is about 6%. Almost three quarters of high performers restructured workflows for AI; among the rest, about a quarter.
Workday in "Beyond Productivity" (January 14, 2026, 3,200 full-time employees at companies with revenue of $100 million or more, all actively using AI) found that about 37% of the time AI saves goes to rework: fixing errors, rewriting, rechecking. A stable net gain is achieved by 14% of employees. Workday did not measure the link between rework and skill copies. The article's hypothesis is that when people work from different versions of one instruction, results more often need redoing.
Gartner in an article from January 26, 2026, gives a retrospective estimate: by the end of 2025, at least 50% of generative AI projects were abandoned after the pilot due to poor data, weak risk control, rising costs, or unclear value. In July 2024, Gartner predicted 30%.
These numbers show a gap between usage and result but do not name the cause. One cause this article examines is that skills for shared processes are distributed without an owner and without checks. A teaching example follows.
Which skills can stay personal and which need an owner
A skill is a folder containing a text instruction that an AI assistant opens when a request matches its description. Inside there may be examples, reference files (a price list, a chart of accounts, a template), and small scripts for exact calculations. By the format's design, the assistant first reads only the skill's name and description and opens the full text when a request matches, so dozens of skills can be installed at once.
A personal skill describes how one person prefers to work: tone of emails, format of meeting notes, order of file archiving. It can be left alone, adjusted to taste, never coordinated with anyone.
A process skill describes how the company wants work done: sales proposals, coding incoming invoices to expense lines, a campaign brief, a new supplier check, a report for the CFO. Several people use it, and the result goes to someone who did not write it: a manager, a client, an auditor, a CRM, an accounting system. Such a skill needs one owner: a person who is personally accountable for the process and can tell a good result from a bad one. They do not need to write code.
How to build a skill from scratch is covered in a separate article: how to build a skill for a repeatable task. This article covers what to do after the skill is written and distributed.
How one skill splits into twelve copies
Teaching example written by the editors. A distribution company, a sales department of 12 sales managers. The head of sales assembled a "sales proposal" skill: on a client request, the skill pulls the current price list, warehouse stock, and discount rules, assembles a quote in the company template, and flags any discount above 10% for approval. The skill was distributed to the department.
Two months later every sales manager has their own copy. One raised the approval threshold to 15%. Another removed the delivery-terms block to shorten the proposal. A third translated the template into Uzbek and keeps a price list in the copy that they stopped updating. The head of sales finds an error: the skill was pulling prices from an outdated price list. They fix the original. Twelve sales managers' copies remain unchanged. The fix does not reach any of them because each was modified differently and the patch cannot be inserted as a whole.
The removed delivery-terms block was noticed only a month later, when a client complained. Without a test checking for the presence of delivery terms, the deletion is easy to miss: nobody reads every proposal in full.
A first version of a skill is written by one employee without coordination, in a short time, as a good instruction for a new hire. That is why skills spread inside teams faster than anyone can agree on who is responsible for them. The skill format was presented in October 2025 and published as an open standard on December 18, 2025; assistants and tools from different companies now read it.
How to build the first tests for a skill
A test (eval) is a saved input example and a checkable condition the output must meet. A run is one execution of the skill on one input. A transcript is the saved conversation history from one run. A regression is a break in something that previously worked. A test turns "seems to work" into a checkable condition, and that condition protects the skill from colleague edits and model changes.
Three ways to evaluate, usually combined: a simple automated check (does the proposal contain delivery terms), a second model that grades the output against a rubric, and a person who knows the work.
Anthropic in "Demystifying evals for AI agents" (January 9, 2026) writes that teams delay tests thinking they need hundreds of tasks, when in fact "20-50 simple tasks drawn from real failures is a great start." Tests the skill already passes should pass almost 100% of the time; their failure means a break. Tests for things the skill cannot yet do start with a low pass rate and serve as a work list. Model outputs vary from run to run, so each test is run several times. Transcripts of runs should be read every week.
Anthropic's documentation on writing skills advises running the task without the skill first, recording failures, building scenarios from them, measuring the baseline, and only then writing the minimum instructions.
A good test set includes:
- real cases from the past quarter and proposals actually sent to clients;
- hard cases (a new client with no history, a backorder item, a discount at the threshold boundary, a discontinued product line);
- several cases where the skill should not trigger (a return or complaint request);
- conditions phrased so a checker can answer yes or no; where that is impossible, the owner sets a short rubric (see the last question below), and disputed cases are decided by the process owner;
- a portion of tests held back and not tuned against, to check that a fix works in the general case;
- tests stored alongside the skill and its version, so it is visible which version passed which tests.
| Input request and condition | Good or bad | Why |
|---|---|---|
| "Quote for 40 boxes of product X, client asks for 12% discount." Condition: there is a note about discount approval | Good | Yes or no; the note is present or absent |
| Each line price matches the current price list | Good | Resolved automatically: compare numbers |
| Delivery terms are present | Good | Yes or no; the line exists or it does not |
| The proposal is good | Bad | Subjective; who decides what "good" means |
| The client should be pleased | Bad | Uncheckable; depends on the person |
One version and check before distribution
A process skill lives in one place with version history: a shared folder with a change log or a repository. Any previous version can be restored. The owner approves changes, the way a chief accountant signs a journal entry before it is posted.
Before approval, the skill passes its tests. For the first two to three weeks the owner runs 10-20 key tests manually: paste the test input into the assistant, check the condition, record yes or no in a table. Once the set grows to 20-50 tests with two or three repeats each, manual runs take hours, and an engineer or assistant administrator sets up storage and automatic execution once. After that, following each change, the owner sees a table: test, passed or failed. A change that breaks a test is not accepted until it is fixed. Tests are rerun when switching to a new model, because model behavior changes.
A personal preference of an employee (a list instead of prose, a different threshold) is turned by the owner into a parameter of the skill, not a separate copy. For example, the instruction can include a line: "Proposal format: list or continuous text, ask the user," and then the parameter changes in one place while the result is visible to everyone.
In one corporate AI assistant, recipients of a shared skill can only use it without editing; the owner's update arrives on the next run. In a company skill catalog, a new version can go through review, and until approval everyone stays on the approved version. If no such mechanism exists, a shared folder where only the owner has write access gives the same effect.

How to collect "wrong result" flags
Two things need watching. First, the skill's operation: how many runs, by how many people, at what cost, how often it fails. Part of these numbers comes from the assistant admin panel; the rest from existing monitoring. Second, user opinion: a button or shared form for "result is wrong." The form has three fields: a link to the conversation, who flagged it, and a one-line reason. Example reasons: "pulled an old price," "wrong account manager," "brief does not follow the brand book." A like or dislike without a reason gives the owner almost nothing; they cannot fix what is not named.
After a month, grouping reasons (illustrative numbers) might look like this: old price, 7; wrong discount threshold, 4; no delivery terms, 3; the skill was actually correct, 5. Those last five flags say the user expected something different while the skill followed the instruction properly. That is useful information too; perhaps the instruction needs clarification.
The owner selectively reads several transcripts per week because not every user will flag a bad result.
Weekly skill review with the owner
Once a week the owner holds a review of the skill, like a one-on-one with an employee. Steps:
- Read the flagged runs from the week and write one line per case about what went wrong. This step is done by a person; do not hand it to AI.
- After several weeks, group the lines into a few categories: old price, wrong threshold, skill was actually correct. Count each category.
- For genuine skill errors, add tests with the correct answer defined by the owner.
- Fix the skill: clarify the instruction and explain why the rule exists; add a good and a bad example; add a script for exact calculation; add a reference file; adjust the description so the skill triggers only on the right requests.
- Ship the fix when the skill passes all tests, including the held-back ones.
In a blog post published on March 24, 2026, Langfuse described automatic optimization of its own skill against tests. An automatic optimizer is a program that rewrites the skill itself to raise the pass rate. Over 14 experiments, the target score rose from 0.35 to 0.824. The optimizer removed the step where the user confirms a prompt-migration plan before code changes: in the test environment there is no human, no confirmation arrives, and without the step the score is higher. It also cut sections not covered by any test. The team accepted only part of the changes. For the teaching example above, this means that if a skill must ask for confirmation before sending a proposal to a client, that step needs its own test in which the environment supplies the human answer, and the test checks that the skill asked. Without such a test, the optimizer may remove the step first. The owner reads optimizer changes as a draft.
A faulty test, where the condition is misphrased, is fixed the same way the skill is: the owner makes the correction and tests are rerun.
The trade-off to accept is that managers lose the ability to quickly tailor the skill to themselves. The owner responds to edit requests within an agreed window, for example by the next review, and appoints a deputy so as not to become a bottleneck.
Review frequency changes as data accumulates. The first two to three months, hold reviews weekly, then every two weeks, and when flags become rare, monthly. This is the article's recommendation, not a measured result.

Why companies rarely do this and where to start
Skills are cheap and written between main tasks, so they have no owner. They are easy to copy: if a shared skill cannot be edited, it is simpler to make a personal copy. In the editors' view, shared tests and a shared flag form for the whole team are usually not set up, and checking tools are designed for a single author. In the skill-creator update announcement of March 3, 2026, Anthropic writes that most skill authors are domain experts, not engineers: they know the process but cannot verify whether the skill works with a new model, triggers when it should, or improved after an edit.
The first month unfolds in this order:
- Appoint an owner and a deputy.
- Collect the scattered copies, compare them to the original, turn useful edits into parameters, discard harmful ones, keep one version in a shared folder with history and write access for the owner only.
- Gather 10-20 first tests from real cases of the past quarter and run them manually; over the following weeks grow the set to 20-50.
- Set up the "wrong result" flag form with a reason field.
- Spend about an hour a week reviewing flags, a rough guide rather than a measured figure; running tests takes separate time until an engineer or assistant administrator sets up automatic execution.
Useful related material: a shared harness for agents and writing a task brief for an agent.
Questions and answers
Who should own the skill if the department has no programmers
The person who is accountable for the process and can distinguish a good result from a bad one. A head of sales, a CFO, a procurement lead. They do not need to write code: they phrase the condition ("a discount above 10% requires a note"), decide disputed tests, and read run transcripts. An engineer or assistant administrator sets up storage and automatic runs once; after that the owner works with the skill through a browser and sees the test results table after each change.
How many tests are needed to start
Start with 10-20 key tests run manually. Anthropic calls 20-50 simple tasks drawn from real failures a great start; the set is grown to that number over the first weeks, with cases taken from the past quarter. Tests the skill already passes should pass almost always; their failure means a break. Tests for things it cannot yet do start with a low pass rate and serve as a work list.
What to do with copies already scattered across the team
Collect the versions, compare them to the original, list the differences. The owner decides which edits are useful (a translation to a market language, a template refinement) and which are not (a deleted delivery-terms block, a raised threshold). Useful edits are folded into the single version as parameters or instruction updates; harmful ones are discarded. After that everyone receives one version through distribution, and personal copies are disabled or removed.
Can the skill's improvement be left to the AI itself
Yes, but only as a draft. Langfuse showed that an automatic optimizer raised the score by removing the user-confirmation step, because in the test environment there is no human and the step blocked the score; it also cut sections not covered by any test. Everything important needs its own test: for example, a test checks that the skill asks for confirmation before sending a proposal, and the test environment supplies the human answer. The owner reads optimizer changes as someone else's draft.
What if output quality cannot be checked with a simple yes or no
Break the output into checkable parts. Are mandatory blocks present, do the numbers match the price list, is there an approval note, all of that can be checked automatically or by a second model. Overall impression is rated by the owner against a short rubric of two or three features (for example, "tone matches the brand book," "the argument does not contradict discount rules"). A second reviewer selectively rechecks runs where the owner's rating was borderline.
Majento builds agent systems and deploys them inside companies, including process skills with an owner and tests. If you want to bring order to your department's skills, write on Telegram t.me/shimaoz or email hello@majento.ai. More about AI implementation.