Inspect AI integration

How to author an eval with Inspect AI, run it, and publish the result to Evaris. This is the primary authoring path today.

Mental model

An Inspect eval is a Task made of four parts:

Inspect produces a .eval log file. Evaris treats that file as the run's artifact. You do not change how Inspect runs; you only publish its output.

The reference example

The walkthrough below evaluates a LangChain support/refund agent with an LLM-as-judge scorer. The layout it uses:

• agents/support_agent.py: the agent under test (LangChain create_agent with order, refund, escalation tools).
• evals/evals.py: wraps that agent as an Inspect solver and defines the task.
• evals/samples.jsonl: the dataset (input + target).

Wrapping an external agent as a solver

The key pattern: your agent is wrapped in a @solver that calls it and writes the result into Inspect's TaskState.output. From evals/evals.py:

from inspect_ai.model import ModelOutput
from inspect_ai.solver import TaskState, Generate, solver

from agents.support_agent import run_agent


@solver
def langchain_agent_solver():
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        del generate
        answer = run_agent(str(state.input))
        state.output = ModelOutput.from_content(model="langchain-agent", content=answer)
        return state

    return solve

The solver is synchronous-ish glue: run_agent invokes the LangChain agent and returns its final message. Inspect handles the rest: dataset iteration, scoring, log writing.

For a fully async agent, make solve await your async call. The shape does not change.

Defining the task

from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_qa

SAMPLES_PATH = Path(__file__).with_name("samples.jsonl")


@task
def support_refund_smoke():
    return Task(
        dataset=json_dataset(str(SAMPLES_PATH)),
        solver=langchain_agent_solver(),
        scorer=model_graded_qa(),
    )

model_graded_qa() is Inspect's built-in LLM-as-judge scorer: it compares the solver's output against the sample's target field and emits a score. Other common scorers: pattern, includes, multi_scorer. See the Inspect scorers docs.

The dataset

JSONL with input and target fields (what json_dataset expects by default):

{"input":"Customer asks for a refund for order A100. What should we tell them?","target":"full refund"}
{"input":"Customer wants refund for order B200 after trying the product. What is the decision?","target":"escalate"}

Running the eval

inspect eval evals/evals.py \
  --model "$MODEL" \
  --log-dir logs \
  --limit 5

• --model is the provider/model string (see OpenAI-compatible providers).
• --log-dir is where the .eval file is written. Publish from this dir.
• --limit 5 caps samples for a smoke run.

Export your provider env vars before running (see the providers guide for the common ZAI_* → OPENAI_* mapping).

Publishing

Once the .eval exists, publish it (see Publish eval logs for details):

evaris publish logs --project <project-id>

What ends up where

Authoring your own eval

1. Put your agent in agents/ (or import it). Expose a single run_agent(question) -> str function.
2. Write a @solver that calls it and sets state.output.
3. Write a @task that wires dataset + your solver + a scorer.
4. Put samples in samples.jsonl.
5. Run with inspect eval ... --log-dir <dir>.
6. Publish with evaris publish <dir>.

For a deeper walkthrough, see Support agent example. For the pattern with no LangChain, see Custom solver.