Evals

An eval is a script of steps that Plico replays against your project, plus what you expect to happen. It needs no model and no API key, so it is a good gate in CI. Evals check your wiring: the tools exist, the approval rules hold, the run ends how you expect. They do not judge what a model would say.

plico eval .

Eval results for .
  PASS refund-approved (eval-refund-approved-5fb6d184)
  PASS refund-denied (eval-refund-denied-12aad571)
       Approval for issue_refund was denied.
  PASS refund-held (eval-refund-held-5607f9f4)

3/3 passed, 0 failed

plico eval exits 0 when every case passes and 1 when a case fails. It also exits 1 when there are no cases, so a CI gate cannot pass with nothing tested. --allow-empty accepts an empty evals/ folder. A new project ships one case, evals/smoke.script.json.

The case fileCopy Anchor Link

Each evals/<name>.script.json file is one case:

{
  "description": "A person approves the refund and the agent confirms it.",
  "script": [
    { "type": "tool.call", "toolName": "issue_refund", "arguments": { "orderId": "A-1001" },
      "result": { "orderId": "A-1001", "status": "refunded" } },
    { "type": "approval.decide", "approved": true },
    { "type": "assistant.output", "content": "Refunded order A-1001." }
  ],
  "expect": { "status": "completed", "outputContains": "Refunded" }
}

The steps are the same ones plico run --dry --script takes, so you can try a case on its own: plico run --dry --script evals/refund-approved.script.json ..

Evals never run tool handlers by surprise. A tool.call step supplies the tool's output as result, and the run uses it instead of calling the handler. A step without result fails the case and names the tool. Add "live": true to run the real handler on purpose. Arguments are still checked against the tool's inputSchema.

Test approvalsCopy Anchor Link

Approval policy applies in evals as it does in a run. Three cases cover a tool that needs approval:

• Held. No approval.decide step. The run stops as waiting and the tool never runs. Use "expect": { "status": "waiting" }.
• Approved. { "type": "approval.decide", "approved": true } right after the tool call. The run records approval.requested and approval.decided, uses the step's result and carries on.
• Denied. "approved": false ends the run as failed before the tool runs. Use "expect": { "status": "failed" }.

An approval.decide step that does not follow a tool call needing approval fails the case. See Approvals.

Turn a run into a caseCopy Anchor Link

plico eval --from-run <run-id> [--name <name>] [path] writes a stored run to evals/<name>.script.json (the name defaults to the run id). Each tool call keeps the output it produced as result, and expect records the run's status and final output. It refuses to overwrite an existing case and fails on an unknown run id. Files a tool wrote are not carried over.

For a complete project with three cases, see Example: a refund agent.