Agent Eval
Compare coding agents head-to-head on your own tasks with pass rate, cost, time, and consistency metrics.
- What
- Compare coding agents head-to-head on your own tasks with pass rate, cost, time, and consistency metrics.
- Cost
- Free
- Needs
- a git repository with real tasks to test, and the coding-agent CLIs you want to compare installed on your machine (e.g. Claude Code, Aider, Codex). The Muse app (mobile or web) for the planning side.
- Level
- Beginner — a git repository with real tasks to test…
- Install
- Copy the installer prompt below into your Muse — your agent does the rest.
Curated by Skill Harbor: a lightweight CLI tool plus method for comparing coding agents head-to-head on reproducible tasks you define in YAML (prompt, files to touch, judge criteria). Judges can be code-based and deterministic (pytest, build), pattern-based (grep), or model-based (LLM-as-judge). Each agent run gets its own git worktree for isolation, no Docker required, and the report shows pass rate, API cost, wall-clock time, and consistency across repeated runs. Best practices included: 3 to 5 real-workload tasks (not toy examples), 3+ trials per agent, pinned commits for reproducibility, at least one deterministic judge per task, and tracking cost alongside pass rate. By @affaan-m, listed here with credit to its creator. From the affaan-m/ECC repository (MIT). Note: the underlying CLI lives in the joaquinhuigomez/agent-eval repository linked by the skill. Honest caveats: developer-oriented, you need a git repo and the agent CLIs you want to compare installed locally. Skill Harbor never reviews the code, review it yourself before use.
Version:
Quick answer
Agent Eval is a free Muse skill that compares coding agents head-to-head on your own tasks, with pass rate, cost, time, and consistency metrics. It uses a lightweight CLI for reproducible tasks defined in YAML, with deterministic, pattern-based, or model-based judges, each run isolated in its own git worktree, and a report on pass rate, cost, time, and consistency.
Install
Copy the install package below, then paste it into Muse🧪 Install ran in our test sandbox · Nov 11, 2026📌 Pinned version: commit 4eb71d9Pinned version: install the code at commit 4eb71d92a39cab44ad40ac9d8a6a5ccb4029d6c2 exactly (Skill Harbor tested this version). Do not substitute a newer version. If this commit is unreachable, stop and tell me. Use "Agent Eval" with your Muse. Prerequisites: a git repository with real tasks to test, and the coding-agent CLIs you want to compare installed on your machine (e.g. Claude Code, Aider, Codex). The Muse app (mobile or web) for the planning side. 1. Open the skill: https://raw.githubusercontent.com/affaan-m/ECC/4eb71d92a39cab44ad40ac9d8a6a5ccb4029d6c2/skills/agent-eval/SKILL.md and copy the full SKILL.md text. 2. Paste it into a chat with Muse and add: "Help me define 3 to 5 benchmark tasks for [my project] in the agent-eval YAML format. Repo at [path], pinned commit [sha]." 3. Install the agent-eval CLI from the repository linked in the skill (review the source first), write your task YAMLs, then run: agent-eval run --task tasks/<name>.yaml --agent claude-code --agent aider --runs 3 4. Ask Muse to help interpret the report table (pass rate, cost, time, consistency) and make the call. Tip: start with tasks from your real workload, not toy examples, and always include at least one deterministic judge (tests or build) per task. Safety: a skill is plain-text instructions; it runs nothing by itself. Review any command before running it, never paste secrets into a chat, and review anything Muse proposes before it acts.
Saved to your recent installs. Find it anytime on /connect.
🔒 Security, provenance and what this touches
The install prompt below already includes the vetting steps: your agent follows the community checklist before installing anything with executable code. Want more?
Verification certificate
Dated facts about what was checked for this listing, and how. This is not a safety certification.
- Pinned version
- 4eb71d9 · 2026-10-13
- Sandbox install test
- Install ran in our test sandbox · Nov 11, 2026
- Creator identity
- Not verified
- Fingerprint watch
- Last checked · Nov 21, 2026 : no change detected in the files linked by this install prompt
- Install prompt analysis
- 2026-10-13 : No commands, network calls, packages, file writes or API keys detected.
Skill Harbor never reviews the code of a listed skill. These are dated facts about the install prompt and our install test, not a safety certification. Review a skill yourself before you install it. How we verify
Questions
How do I install a skill?
Every product page includes a copy-paste install prompt. Paste it into your Muse and it sets the skill up for you — no manual configuration.
Where does my money go?
Straight to the seller. Skill Harbor never processes payments: checkout happens on the seller’s own page, usually Stripe.
What does the ✓ next to a creator’s name mean?
It means we confirmed the identity of the person behind the listing. It says nothing about the code itself — always check a skill before installing it.