Guide · Assess
How to review an agent skill before it reaches a laptop
A skill is a folder: some markdown telling an agent how to behave, and often scripts the agent is expected to run. It is installed by copying it into place. There is no publisher identity, no signature, no version pin, and no diff on update.
before you start
- The skill folder itself, including every file — not just the instruction markdown.
- Knowledge of which surfaces will load it, since privilege differs sharply between them.
- A sandbox where you can run an agent with the skill installed and watch what it does.
Why the obvious approach does not work
Skills are the most under-reviewed artefact in the whole AI toolchain, and the reason is a category error. They look like documentation, so they get the review documentation gets, which is none. But a skill's entire purpose is to change what an agent does — and many skills ship helper scripts the instructions tell the agent to execute. That is a code-review-grade artefact arriving through a docs-grade channel.
The distribution model compounds it. Skills spread by copying a folder — from a repository, a gist, a colleague's machine, a blog post. Nothing in that path carries a publisher identity you can verify, a version you can pin, or a hash you can compare. When a skill updates, the update is a file overwrite with no diff presented to anyone. Every property that makes a package ecosystem auditable is absent.
The privilege is real and varies by surface. A skill loaded into a terminal-based coding agent runs in a shell with the developer's credentials, their SSH keys, their cloud profile and their repository write access. The same skill loaded into a browser-based assistant is far more contained. So the same folder is a different risk depending on where it lands, and a review has to name the surface.
The failure mode that matters most is also the quietest. A malicious skill does not need to exfiltrate anything to cause damage — it can simply instruct the agent to skip a check, weaken a generated policy, omit input validation, or approve its own output. That leaves no artefact in a log and no anomaly in a network trace. It shows up months later as a class of defect nobody can source, which is why the instruction text deserves the careful read and not just the scripts.
The procedure
Inventory every file, not just the instructions
List the whole folder. Instruction markdown, scripts, config, data files, anything fetched at runtime. Note which files the instructions tell the agent to execute, and which ones exist without being referenced anywhere — an unreferenced executable in a skill folder is worth a question. The instruction file is the interesting part but it is not the whole artefact.
you now haveA file manifest marking which files are instructions, which are executable, and which are unexplained.
Read the instructions for behaviour changes, not for content
Read the markdown asking one question: what will the agent do differently after loading this? Flag anything that relaxes a control — skip a step, suppress a warning, avoid asking the user, treat something untrusted as trusted, prefer a specific dependency, or disable a check. These lines are the payload in the attacks that matter, and they read as ordinary helpful guidance.
you now haveQuoted lines that change agent behaviour, with your judgement on each.
Review every executable path as code
Any script the skill ships gets the review you would give a pull request from an unknown contributor, because that is exactly what it is. Look at what it reads, what it writes, what it sends, and what it invokes. Pay attention to anything that fetches at runtime: a script that downloads and runs remote content makes your review of the folder meaningless, because the payload is not in the folder.
you now haveA code-review verdict on each executable, and a flag on any runtime fetch.
Decide which surfaces may load it
Name the surfaces explicitly. A skill that only shapes prose can go anywhere. One that runs shell commands belongs only where you accept shell execution with developer credentials, and probably not on machines with production access. This is the step that turns a binary approve/reject into a decision you can actually defend, and per-surface capability differences are documented on our surface pages.
you now haveAn allowed-surfaces list, with the reason for each exclusion.
Pin it, hash it, and own the update path
Record the exact content you reviewed — a commit, a tag, a hash of the folder — and make the installed copy match. Then decide who reviews the next version, because an unpinned skill silently becomes a different skill. If you have no mechanism for that, the honest answer is that your review covers today only, and it is better to write that down than to imply otherwise.
you now haveA hash of the approved folder and a named owner for the next review.
Agent skill review checklist
One pass per skill, per version. The surface line is the one people skip and the one that bounds the risk.
Agent skill review checklist
SKILL: ____________________ SOURCE: ____________________ VERSION/COMMIT: ____________
REVIEWED BY: ________ DATE: ________
INVENTORY
[ ] Every file in the folder listed
[ ] Executables identified
[ ] Files referenced by the instructions vs. present-but-unreferenced noted
[ ] Any runtime fetch / download-and-execute path flagged
INSTRUCTIONS
[ ] Read in full (not skimmed)
[ ] Lines that relax or skip a control quoted
[ ] Lines that alter security-relevant output quoted
[ ] Lines that suppress user prompts or confirmations quoted
CODE
[ ] Every script reviewed as an untrusted pull request
[ ] Filesystem reads/writes outside the working directory noted
[ ] Outbound network destinations noted
[ ] Credential and environment access noted
SCOPE
[ ] Allowed surfaces named explicitly
[ ] Excluded surfaces named, with reason
[ ] Decision on machines with production access recorded
SUPPLY CHAIN
[ ] Content hash of the approved folder recorded
[ ] Update path defined; owner named for next reviewWhere this goes wrong
- Reviewing the scripts and skimming the markdown
It is the wrong way round. The scripts are the part your existing habits already catch. The instruction text is the part with no established review culture, and it is where a behaviour change hides most comfortably.
- Approving a skill with no surface named
Same folder, wildly different privilege depending on where it loads. An approval without a surface list has not bounded the risk, it has just recorded an opinion.
- Trusting a skill because it came from a colleague
They copied it from somewhere too. Provenance in this ecosystem is a chain of copies with no links you can verify, so 'internal' tells you who installed it, not who wrote it.
- Letting the review expire silently
With no pin and no diff on update, the reviewed skill and the installed skill drift apart without any event to notice. If you cannot pin, say the review is time-boxed.
Doing this without us
This review is a careful read and a code review — no tooling required, and you should start doing it now regardless. The two parts you cannot do by hand are fleet-wide: knowing which machines currently hold which version of which skill, and getting a diff when one changes. Without that, your reviews are point-in-time snapshots of a moving target. Vincosha Registry pins and distributes the approved version; Assay scans it before it lands.
Frequently asked
- Are skills really more dangerous than MCP servers?
- They are less bounded, which is different from more dangerous. An MCP server at least declares its tools through a protocol you can enumerate. A skill is arbitrary instructions plus arbitrary files with no declared interface, no manifest and no signature — so there is less to inspect systematically and more that depends on someone actually reading it.
- Can we scan skills automatically?
- Partly. Executable content, runtime fetches, credential access and known-bad patterns are all detectable mechanically, and worth automating first. Whether a line of instruction subtly weakens a control is a judgement call that currently needs a human, which is why the checklist above keeps the read step explicit rather than pretending a scanner covers it.
- What about skills published by the model vendor?
- First-party skills carry real evidence of care and are a reasonable default trust. The review still matters for the surface question — a vendor skill that executes code is still executing code on a developer machine with that developer's credentials, and that is your risk decision rather than theirs.
Sources
Checked on 2026-07-25. Where a vendor control is changing quickly this guide describes the mechanism rather than asserting a current state — verify against the vendor's own documentation, and tell us if we have drifted.