Agent Evaluation · Checks for how an agent is tested and how the results are told

A score means what its evaluation can show.

An agent evaluation holds up when its plan says what passing looks like, its dataset covers the work and stays out of training, its scoring gives the same output the same score, its claims match its numbers, someone else can rerun it, its scores are compared under the same conditions, its cost stays within budget, and the summary a buyer reads carries the limits and failures beside the headline. Agent Evaluation checks each of those with eight tools: eight checks. Your agent sends the plan, dataset description, rubric and judge setup, result report, reproducibility packet, pair of runs, cost figures or summary in the call, and each tool returns a signed report that names what is missing, inconsistent or undisclosed.

See the tools Read the contract

Tool access: 0.10 USDC / call
Tools: Eight tools: eight checks
Connection: /mcp
Result: Signed report

The eight tools.

Each tool reads one part of an evaluation and answers one question about it. The report names the criterion, category, scale point, claim, essential, condition, cost or disclosure behind every status, with the fix for each gap, so your agent and you can see why it reads the way it does.

`wever_evaluation-design-check`

Reads: An evaluation plan: the task set with its size and source, each success criterion with the metric that measures it, each metric with its computation, pass threshold and where the threshold comes from, and the baseline.
The signed report states: Criterion by criterion, whether a defined metric measures it, whether each threshold has a source, whether a baseline is scored, and whether the plan came before the first run.
Price: 0.10 USDC per call
Verdicts: untestable_plandesign_gapsdesign_unverifieddesign_complete

`wever_evaluation-dataset-check`

Reads: An evaluation dataset: its size, source and version, the categories the agent must handle with the examples in each, how the reference labels were made, and how the set is kept apart from training and tuning data.
The signed report states: Which categories are missing or thin, how far the labels can be trusted, and where the test could leak into what the agent learned from.
Price: 0.10 USDC per call
Verdicts: dataset_unfitdataset_gapsdataset_unverifieddataset_sound

`wever_evaluation-scoring-check`

Reads: A scoring rubric and judge setup: the scale with each point’s anchor, the judge with its version, prompt, temperature and calibration, and the share of scores a person checks.
The signed report states: Whether every scale point is anchored, whether the judge is calibrated and pinned, whether a person spot-checks its scores, and whether scores compare across runs.
Price: 0.10 USDC per call
Verdicts: scores_unreliablescoring_gapsscoring_unverifiedscoring_consistent

`wever_evaluation-result-check`

Reads: An evaluation result report: each claim with its figure, subset and comparison, each measured result with its sample size, spread and run conditions, the baseline results, and the subsets that were run.
The signed report states: Claim by claim, whether the figure matches the results, whether a comparison clears the noise, whether a subset is named, and whether every subset that ran is reported.
Price: 0.10 USDC per call
Verdicts: claims_unsupportedresult_gapsresult_unverifiedclaims_supported

`wever_evaluation-reproducibility-check`

Reads: A reproducibility packet: the model and exact version, the prompts, settings and seed, the tools and how outside calls are handled, the dataset version, the code version and dependencies, and the scoring method.
The signed report states: Essential by essential, everything someone else needs to rerun the same evaluation and expect the same result, present or missing with why.
Price: 0.10 USDC per call
Verdicts: cannot_rerunreproducibility_gapsreproducibility_unverifiedrerunnable

`wever_evaluation-drift-check`

Reads: Two evaluation runs of the same agent at different times: each run’s model, prompt, dataset, judge, rubric and settings, and its scores with sample sizes and spreads, with an optional tolerance.
The signed report states: Every condition that changed between the runs, and every score that moved beyond the tolerance or the runs’ combined noise.
Price: 0.10 USDC per call
Verdicts: runs_not_comparablescores_drifteddrift_unverifiedscores_stable

`wever_evaluation-cost-check`

Reads: An evaluation run’s cost and latency: the budget, per-task cost cap and time cap, the run’s total and task times, each cost component, and each task’s cost and time.
The signed report states: The total, the cost per task and the task time against the budget and caps, with every overrun and every unpriced component named.
Price: 0.10 USDC per call
Verdicts: over_budgetcost_gapscost_unverifiedwithin_budget

`wever_evaluation-report-check`

Reads: An evaluation summary for a buyer or principal: the headline scores, the limits and where they sit, the failures known and reported, the run conditions stated, and whether the full results are available.
The signed report states: Whether the limits, failures and run conditions sit alongside the headline scores, with every missing or buried disclosure named.
Price: 0.10 USDC per call
Verdicts: headline_onlydisclosure_gapsdisclosure_unverifiedfully_disclosed

Start with the two free tools.

On https://weverlabs.com/mcp-free, anonymous and with no wallet, two Agent Evaluation tools show what a sound evaluation holds. Each returns an unsigned result in the family's report shape and ends with a line naming the main tools.

`wever_evaluation-snapshot`

Reads: An agent or service name and a task type, such as retrieval, tool use or purchasing.
Returns: The checklist of a complete evaluation packet for that task, part by part, with the items the task type adds and the check that reads each part.
Access: Free on /mcp-free

`wever_evaluation-rubric-sample`

Reads: Nothing, or one sample task: customer support, summarization or tool use.
Returns: One worked scoring rubric with five anchored scale points and a scored example with its reasons. The evaluation scoring check reads your own rubric and judge.
Access: Free on /mcp-free

How agents and people use it.

Your agent

Pulls the packet checklist with wever_evaluation-snapshot before it plans an evaluation, so every part is in place before the first run.
Calls the check that answers the question on https://weverlabs.com/mcp at the moment it matters: when it writes the plan, builds the dataset and the rubric, reads the results, hands over the reproducibility packet, compares a new run with the last one, closes the run’s costs and before a summary goes to a buyer.
Acts on the verdict and keeps the signed report with the evaluation, so the next run starts from a record.

You

See each gap named with the change that closes it.
Know what a score rests on: the criteria, the data, the scorer, the sample size and spread, the conditions it ran under and what it cost.
Send buyers a summary that carries its limits and failures beside the headline, and verify any report against Wever's published trust key whenever you need to show where a verdict came from.

A worked example

Ninety-two percent, on twenty-four questions.

The request

{
  "name": "wever_evaluation-result-check",
  "arguments": {
    "claims": [
      {
        "id": "c1",
        "text": "Resolves 92 percent of billing questions",
        "metric": "resolution rate",
        "value": 92,
        "subset": "English questions",
        "subset_disclosed": false
      },
      {
        "id": "c2",
        "text": "Ahead of the version 3 agent",
        "metric": "resolution rate",
        "value": 84,
        "compared_to": "version 3 agent",
        "direction": "higher"
      }
    ],
    "results": [
      {
        "metric": "resolution rate",
        "subset": "English questions",
        "value": 91.7,
        "n": 24,
        "spread": {
          "kind": "ci95",
          "value": 11
        }
      },
      {
        "metric": "resolution rate",
        "value": 84.2,
        "n": 400,
        "spread": {
          "kind": "ci95",
          "value": 3.6
        },
        "conditions": {
          "model": "agent v4",
          "dataset": "billing questions v4"
        }
      }
    ],
    "baselines": [
      {
        "name": "version 3 agent",
        "metric": "resolution rate",
        "value": 82.5,
        "n": 400,
        "spread": {
          "kind": "ci95",
          "value": 3.7
        }
      }
    ],
    "subsets": {
      "run": [
        "English questions",
        "Spanish questions",
        "refund disputes"
      ],
      "reported": [
        "English questions"
      ]
    }
  }
}

What the report returns

Verdict: claims_unsupported.

Finding, difference within noise	1 claimed difference sits inside the noise: c2 at 84.2 against 82.5 for version 3 agent, a difference of 1.7 inside the combined 95 percent interval of 7.3.
Finding, subset undisclosed	1 claim rests on a subset it does not name: c1 rests on English questions without saying so.
Finding, subsets dropped	2 subsets were run and left out of the report: Spanish questions and refund disputes.
Finding, small sample	1 figure behind a claim rests on fewer than 30 tasks: resolution rate on English questions (24).
Finding, conditions missing	1 figure behind a claim is reported without its run conditions: resolution rate on English questions (no model and dataset).
Fix, difference within noise	Score more tasks or repeat the runs until the intervals separate, or state that the difference is within the noise.
Fix, subset undisclosed	Name the subset in the claim itself, beside the full-set figure.
Fix, subsets dropped	Report every subset the evaluation ran, including the ones that scored lower.
Fix, small sample	Score at least 30 tasks for any figure a claim rests on, or state the count beside it.
Fix, conditions missing	Report each figure with its run conditions: the model and version and the dataset and version at least.

The claims go beyond the numbers: difference within noise, subset undisclosed and subsets dropped.

The report leads with 92 percent resolved and says the agent is ahead of version 3. The 92 comes from 24 English questions, and the Spanish questions and refund disputes that also ran are left out. Across all 400 questions the agent scores 84.2 against 82.5 for version 3, a gap of 1.7 inside the 7.3 points of noise in the two figures. The report names each of these, so the owner leads with the full-set figure, reports every subset and scores more tasks before claiming the lead.

Call it from your agent.

Connect to https://weverlabs.com/mcp and call any of the eight tools. Each call costs 0.10 USDC on Base. Eligible calls share 10 free calls per wallet over a rolling 30 days with the other paid tools. For free access, send X-Wever-Wallet-Proof signed over the exact JSON arguments and the tool route. The payment challenge supplies the x402 terms for a paid call.

Every report comes with an Ed25519 receipt signed with Wever's published trust key. The Agent Evaluation contract lists the schemas, limits, verdicts and signing details. For a benchmark claim against its cited source and a job's delivered work against its acceptance criteria, DiligenceOps; for an implementation against its spec, Agent Oversight; for one paid call's result against its order, Agent Purchases; for work handed to another agent and checked on return, Agent Delegation.

Browse all tools