Lab: Measure Whether an Agent Skill Actually Helps

• Design a fair baseline versus with-skill comparison on ten tasks.
• Record pass/fail, wall time, tokens and human score for each run.
• Compute cost per successful task and decide whether the skill earns its context and maintenance cost.

• A coding agent you can run non-interactively, and one skill or pack to test (your own, or one from the catalogue that you have read and pinned).
• A repository with a working test suite, so most tasks have an automatic pass check.
• Basic shell and git. Optionally Node or Python for analysis.
• Familiarity with the skills and harnesses guide.

1. Create a separate directory for the experiment. Do not run it inside a repository you care about.
2. Use a disposable clone or container per run (the script skeleton does this) so state cannot leak between runs.
3. Pin: the model name and version, the agent client version, and the skill's commit hash. Write them in the results file header.
4. Use a scoped, low-limit API key or a local model; set a spend cap in the provider console before you start.

1. Define ten tasks. Mix sizes: bug fixes, small features, refactors, a documentation task. Each needs an unchanged prompt and a pass check. Write them using the template below before you run anything, so you cannot tune tasks to the result.
2. Decide the hypothesis. For example "the skill raises pass rate or cuts human edits without more than X% extra cost". Write X down in advance.
3. Run the baseline. Same model, same settings, no skill. Record the outcome of each run.
4. Run with the skill. Identical in every way except the skill is present. Randomise order or alternate to avoid time-of-day or rate-limit effects.
5. Repeat. Agent runs are non-deterministic. Run each task at least three times per condition if budget allows.
6. Record tokens from the agent's own usage output (most clients print or log input and output token counts). Do not estimate them.
7. Score by hand. Apply the rubric to each result, blind to the condition if possible.
8. Price it. Use the LLM cost calculator for your model, or the formula below.
9. Decide. Compare pass rate, cost per success, wall time and human score. State the result, including when it is inconclusive.

Task list template

# tasks.md  (one block per task; keep them realistic and checkable)

## T01
Repo/commit: <path or commit hash the task starts from>
Prompt: <exact text you will give the agent, unchanged between conditions>
Pass check: <command that exits 0 on success, e.g. "npm test -- foo.test.ts">
Notes: <what a human would look for that the check cannot see>

Results CSV header

task_id,condition,run,passed,wall_seconds,input_tokens,output_tokens,human_score,notes
T01,baseline,1,,,,,,
T01,with_skill,1,,,,,,

Scoring rubric

Human score (0-3), assigned blind to the condition if you can:
0 = wrong or unusable
1 = partly right; needs substantial rework
2 = correct; minor edits (naming, style)
3 = correct and ready to merge as is
Also note: unrequested changes (scope creep), skipped checks, invented facts.

Run script skeleton

This is a skeleton, not a working tool. AGENT_CMD, task-field.sh and the token parsing are placeholders for your client and task format. Read your client's documentation for its non-interactive mode and usage output.

#!/usr/bin/env bash
# run-one.sh <task_id> <condition> <run>
# SKELETON. Replace AGENT_CMD with your client's non-interactive invocation (see its docs).
set -euo pipefail
task="$1"; cond="$2"; run="$3"
work="$(mktemp -d)"                       # fresh checkout every run: no contaminated state
git clone --quiet "$REPO_URL" "$work/repo"
( cd "$work/repo" && git checkout --quiet "$(./task-field.sh "$task" commit)" )
if [ "$cond" = "with_skill" ]; then cp -R ./skill-under-test "$work/repo/.claude/skills/"; fi
start=$(date +%s)
# AGENT_CMD should print its own usage summary (tokens) to a log you can parse.
( cd "$work/repo" && AGENT_CMD "$(./task-field.sh "$task" prompt)" > "$work/agent.log" 2>&1 ) || true
end=$(date +%s)
if ( cd "$work/repo" && eval "$(./task-field.sh "$task" check)" ) >/dev/null 2>&1; then passed=1; else passed=0; fi
echo "$task,$cond,$run,$passed,$((end-start)),<in_tokens>,<out_tokens>,," >> results.csv
rm -rf "$work"

Cost formula

cost_per_run = (input_tokens / 1e6) * input_price_per_million
             + (output_tokens / 1e6) * output_price_per_million
cost_per_success = total_cost_for_condition / number_of_passed_runs
total_experiment_cost = cost_per_run_avg * 10 tasks * 2 conditions * runs_per_task

Prices change; take current per-token prices from your provider or the calculator rather than from this page.

• A results CSV with one row per task, condition and run.
• A summary table per condition: pass rate, mean wall time, mean tokens, mean human score, cost per successful task.
• A one-paragraph conclusion that states the hypothesis, the result and its limits.

• Both conditions used identical tasks, model, settings and fresh working state.
• Each task was run more than once per condition, or you state that you did not and treat the result as anecdotal.
• Token counts come from the client's output, not estimates.
• Pass checks were written before the runs.
• Your conclusion does not claim more than ten tasks can support.

• Non-determinism. One run per cell proves little. Repeat and report the spread.
• Contaminated context. Leftover files, memory, caches or earlier session state favour one condition. Use a fresh clone and a new session each time.
• Unfair comparison. Different prompts, models, order effects, or tuning the skill on the same tasks you then measure. Hold out tasks you did not use while writing the skill.
• Weak checks. A passing test may not mean a good change. Keep the human score.
• Hidden costs. Count the skill's standing context cost and your maintenance time, not only per-run tokens.

• Delete temporary clones and containers.
• Revoke or rotate the API key you used for the experiment.
• Keep the CSV and the pinned versions in a repository so the result can be reproduced or challenged.

• Read the skill and any scripts it contains before the first run.
• Run with the network restricted where the tasks allow.
• Use a throwaway key with a hard spend limit.
• Do not commit transcripts without checking them for secrets and private code.

Duration (estimate): roughly 3 to 5 hours, assuming ten small tasks, a working test suite and non-interactive runs that each take a few minutes. Task writing and scoring dominate the time.

Cost: use the formula above. The number of runs is 10 tasks times 2 conditions times runs per task, and the per-run token counts and prices come from your own measurements and provider. We do not state a dollar figure because it depends on your model and tasks. Run one task of each condition first, price it, then multiply.

• Local model. Run an open-weight model through Ollama (see Local AI) with a client that can use it. Small local models may follow skills less reliably, which is itself a finding. Cost is local compute only; record wall time and token counts from the runtime.
• Score existing transcripts. If you already have agent sessions with and without a skill, apply the rubric to those transcripts and compare. This is weaker evidence because tasks and conditions were not controlled, so say so.