Custom solver · Evaris Docs

The LangChain example (previous page) is one instance of a general pattern: wrap any callable in an Inspect @solver and score the result. This page shows the pattern stripped down, for an arbitrary agent function.

The minimal solver

from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.model import ModelOutput
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import TaskState, Generate, solver

# Your agent: any function that takes a prompt and returns a string.
# Could be a hand-rolled loop, an SDK call, an HTTP call to a service, etc.
def my_agent(prompt: str) -> str:
    # ... your logic ...
    return "agent reply"


@solver
def my_agent_solver():
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        del generate
        answer = my_agent(str(state.input))
        state.output = ModelOutput.from_content(model="my-agent", content=answer)
        return state
    return solve


@task
def my_task():
    return Task(
        dataset=json_dataset("samples.jsonl"),
        solver=my_agent_solver(),
        scorer=model_graded_qa(),
    )

That is the entire bridge. Inspect handles dataset iteration, scoring, and log writing.

Async agents

If your agent is async, await it inside solve:

async def my_agent_async(prompt: str) -> str: ...

@solver
def my_agent_solver():
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        del generate
        answer = await my_agent_async(str(state.input))
        state.output = ModelOutput.from_content(model="my-agent", content=answer)
        return state
    return solve

Multi-turn / tool-using agents

Inspect's TaskState carries the full message history in state.messages. For an agent that should see prior turns (e.g. when inspect itself is driving tool calls), use generate(state) instead of ignoring it:

async def solve(state: TaskState, generate: Generate) -> TaskState:
    # inspect runs the model forward, the solver can inspect/modify state,
    # then hand back via generate.
    return await generate(state)

That delegates to Inspect's built-in solving. See the Inspect solvers docs for the full state machine; most users with their own agent will want the simple "call my function, write the output" shape from the first example.

Choosing a scorer

Scorer	When
`model_graded_qa()`	Free-text answers judged against a rubric (`target`). The default for agent evals.
`pattern(r"\b\d+\b")`	Output must match a regex (e.g. extract an amount).
`includes("refund")`	Output must include a substring.
`multi_scorer(...)`	Several scores per sample.

See Inspect scorers.

Running + publishing

Same as the LangChain example:

inspect eval my_eval.py --model "$INSPECT_EVAL_MODEL" --log-dir logs
evaris publish logs --project <project-id>

The publishing side does not care how your solver was built; it only sees the .eval file Inspect produced.