agent

Every agent call is one bounded invocation: a fresh observation, an explicit deadline, a model-call budget, and no shared transcript between calls. Acquiring the fixture without a configured model fails with MODEL_UNAVAILABLE.

export interface Agent {
  act(instruction: string, options?: ActOptions): Promise<ActResult>;
  waitFor(condition: string, options?: WaitForOptions): Promise<void>;
  extract<Schema extends StandardSchemaV1>(
    instruction: string,
    options: ExtractOptions<Schema>,
  ): Promise<StandardSchemaV1.InferOutput<Schema>>;
  assert(assertion: string, options?: AssertOptions): Promise<void>;
}

Budgets

Model calls count every model request including repair rounds. Action steps count committed engine actions. Every timeout is additionally capped by the remaining test timeout.

Method	Model calls	Action steps	Default timeout
`act`	`agents.<name>.maxModelCalls`, 25	`agents.<name>.maxSteps`, 25	`config.timeout`, 120000 ms
`waitFor`	`agents.<name>.maxModelCalls`, 25	0	the judgment budget
`extract`	2	0	the judgment budget
`assert`	2	0	the judgment budget

The judgment budget is max(30000, config.actionTimeout) milliseconds. Per-call budgets must be positive integers and can only lower the configured limits. Every duration option is milliseconds. The assert row holds for the built-in agent. When the selected agent has a custom executor, assert is dispatched to it as a step: its budget is the agent’s maxSteps and maxModelCalls, its default timeout is config.timeout, and vision or screenshot on the call is UNSUPPORTED_CAPABILITY.

agent.act

act(instruction: string, options?: ActOptions): Promise<ActResult>;

Plans and executes one bounded multi-action flow. A passing verdict resolves with an ActResult; a failing or blocked one throws an AgentError whose code and blocked flag say why.

Name	Type	Default	Valid
`instruction`	`string`		1 through 8192 UTF-8 bytes after NFC
`options.params`	`AgentParams`	`undefined`	JSON-safe values plus `Secret`, at most 64 KiB and 32 levels deep
`options.timeout`	`number`	`config.timeout`	positive safe integer
`options.maxSteps`	`number`	the agent’s `maxSteps`	positive integer, at most the resolved limit
`options.maxModelCalls`	`number`	the agent’s `maxModelCalls`	positive integer, at most the resolved limit
`options.agent`	`string`	the test’s agent	a name in `agents`; unknown is `INVALID_ARGUMENT` before the step opens

Values come from your code through options.params, never from the model. A Secret in params is projected before the model sees anything: the model receives the secret’s name and purpose, and the harness fills the value host-side under the same authorization as locator.fill: the credential must be registered, the current origin allowed by both the target and the credential, and the field an enabled secure input whose purpose matches.

await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });

const member = credentials.user('member');
const { summary, cache } = await agent.act('Sign in', {
  params: { username: member.username, password: member.password },
});

The result carries the step’s summary, the modelCalls and actions it spent, and cache: how the trace cache took part (self-finalized, agent-concluded, or missed with a reason; absent when caching is off).

Pixel tools: screenshot and tap_at

act takes no vision option. Pixels reach an act step through two tools the built-in agent offers while pixels can leave the runner. The act model itself receives the image: no separate vision model, no extra model call, and nothing lost in a description.

Tool	What it does
`screenshot()`	Attaches a masked screenshot of the viewport to the tool result, with the changes since the screen the model holds. From then on the step is in pixel mode: every action result carries a fresh screenshot, so the model sees what each action did.
`tap_at({ x, y })`	Taps a point given in the latest screenshot’s pixel coordinates. The harness scales the point into the observation’s CSS pixels and hit-tests it against the tree: on an engine with `perform`, a listed control under it is tapped by its id, exactly like `tap`, with a replayable descriptor; a point on nothing listed, or any point on an engine that taps by point only, is tapped as a bare point through the engine’s `tapAt`. The result names what was tapped or what sits under the point, and carries a fresh screenshot.

While the step is showing pixels, the wait after an action watches the pixels too: a tap that redraws a canvas is read as soon as the redraw lands, instead of waiting out the change window on a tree that never moves. A screen that lists nothing to act on by id (a canvas, a game, a semantics-free native screen) opens with a screenshot already attached, so the model never spends a turn discovering that it needs one; any screen with a listed control opens tree-only and the model asks when the tree falls short. Older screenshots are elided from the conversation in batches, keeping the newest two, so a long flow in pixel mode carries a bounded number of images, and each is resampled to a long side of at most 768 pixels before it goes to the model: on a drawn keypad and a drawn wizard that size kept every run passing at a third less cost than the full capture, and 512 did not. Coordinates the model returns are in that image; the harness scales them back. Both tools are offered while no secret has been filled when the step opens; a secret filled during the step keeps them listed but withholds every later screenshot (PIXEL_TAINTED), and each result says so; tap_at needs an engine that can tap by node (perform) or by point (tapAt, the pointer capability, which the Playwright and agent-device engines declare), and on an engine without tapAt the point must land on a listed control. A bare-point tap replays from the trace cache the way a coordinate-driven tool does: the same point on a viewport of the recorded size, or, when a listed node contained the point, the same place inside that node’s live box. A viewport of another size or a node that cannot be re-found hands the step to the model, and the recorded end state still has to return before a replay passes on its own. A step that sent pixels records visionInput: true and metrics.pixelBytes. The pixel path asks the act model to ground a point in an image, and models differ widely there. Measured on a drawn ten-key keypad and an eight-screen drawn wizard: the gpt-5.6 luna family and Gemini 3.8 Flash completed both; Claude Sonnet 5 completed the wizard’s large shapes but never the keypad’s small keys; DeepSeek and GLM flash models gave up, and their providers bill a 1280 by 720 screenshot at tens of thousands of input tokens against a few hundred on OpenAI. Pin an act model that grounds well when your flows leave the tree. Throws ACTION_FAILED for a failed verdict with no more specific code; a blocked code (AUTH_CREDENTIAL_UNAVAILABLE, ENVIRONMENT_UNAVAILABLE, SEED_DATA_MISSING, TEST_SETUP_FAILED, AUTOMATION_UNSUPPORTED) when the step could not reach a product verdict; STEP_BUDGET_EXHAUSTED and STEP_TIMEOUT when a budget or the deadline ran out; INVALID_ARGUMENT for a malformed instruction, oversized params, or an unknown option; and UNSUPPORTED_CAPABILITY for schema or vision, which act does not take.

agent.assert

assert(assertion: string, options?: AssertOptions): Promise<void>;

One fresh observation, exactly one judgment. It does not poll.

Name	Type	Default
`assertion`	`string`
`options.timeout`	`number`	`max(30000, config.actionTimeout)`
`options.screenshot`	`boolean`	`true`
`options.vision`	`VisionMode`	the agent’s `vision`, `false`
`options.agent`	`string`	the test’s agent

Throws ASSERTION_FAILED on a false judgment, carrying the explanation and, when permitted, a screenshot path. Evidence is skipped with screenshot: false and once any secret has been filled in the attempt. screenshot controls failure evidence attached after the judgment; vision controls what the model is shown before it.

await agent.assert('the dashboard shows a trial badge');

agent.waitFor

waitFor(condition: string, options?: WaitForOptions): Promise<void>;

One observation per interval, and a judgment when the screen changed. Succeeds on the first true judgment.

Name	Type	Default	Valid
`condition`	`string`
`options.timeout`	`number`	`max(30000, config.actionTimeout)`	positive safe integer
`options.interval`	`number`	`3000`	integer 100-60000, milliseconds
`options.maxModelCalls`	`number`	the agent’s `maxModelCalls`	positive integer, at most the resolved limit
`options.vision`	`VisionMode`	the agent’s `vision`, `false`
`options.agent`	`string`	the test’s agent	a name in `agents`

Throws STEP_TIMEOUT on deadline expiry and STEP_BUDGET_EXHAUSTED on budget exhaustion, each carrying the last judgment, and INVALID_ARGUMENT for an out-of-range option or one waitFor does not take.

await agent.waitFor('the upload progress reports completion', { interval: 500 });

agent.extract

extract<Schema extends StandardSchemaV1>(
  instruction: string,
  options: ExtractOptions<Schema>,
): Promise<StandardSchemaV1.InferOutput<Schema>>;

Takes one fresh observation, requests structured output, validates it, and returns the validated value. It performs no app actions.

Name	Type	Default	Valid
`instruction`	`string`
`options.schema`	`Schema`		any Standard Schema v1 validator
`options.timeout`	`number`	`max(30000, config.actionTimeout)`	positive safe integer
`options.vision`	`VisionMode`	the agent’s `vision`, `false`
`options.agent`	`string`	the test’s agent	a name in `agents`

The budget is two model calls: one extraction and one repair round for output that failed validation. Throws INVALID_ARGUMENT for a schema that is not Standard Schema v1, and MODEL_OUTPUT_INVALID when validation still fails after the repair.

const summary = await agent.extract('the workspace name and plan', {
  schema: z.object({ name: z.string(), plan: z.string() }),
});

Vision

assert, waitFor, and extract accept vision. It controls what evidence the model is shown. Sending pixels needs a model that accepts images; images are billed as input tokens. act has no such option; its pixel path is the screenshot and tap_at tools above.

Mode	What the model sees	Cost
`false`	the tree	no image tokens
`true`	the tree and a masked screenshot, every call	image tokens on every call
`'only'`	the screenshot, and not the tree	image tokens, minus the observation bytes

Set vision on the agent in config to make a mode the project default; a per-call value wins. Pixels go to the agent’s model, the same one the act loop shows its screenshots to.

Use 'only' for questions about what the page presentsvision: true sends the tree and the screenshot, and the tree is a cheaper path to an answer. A consent sheet covering a form, a price at opacity: 0, a button under a cookie banner: the tree reports all of them as fine.

// Passes on the tree alone while a consent sheet covers the form.
await agent.assert('the search form is visible and not covered', { vision: true });

// Judged on the screenshot. There is no tree in the request to answer from.
await agent.assert('the search form is visible and not covered', { vision: 'only' });

Use true when the tree is part of the answer, such as a disabled state that is not visible at all.

The screenshot is the viewport, never the full page, so 'only' cannot see below the fold. Write 'only' conditions about what is on screen and let the flow scroll first. Pixel evidence degrades instead of failing the call in every mode but 'only'. It is withheld, and the tree sent alone, when a secret fill has tainted the viewport (visionDegraded: "PIXEL_TAINTED" on the step), when the engine masked fewer regions than the secure fields it observed (MASKING_UNPROVEN), or when the engine cannot capture pixels (UNSUPPORTED_CAPABILITY). 'only' fails with POLICY_DENIED in those cases rather than answer a question about pixels from the tree. A step that sent pixels records visionInput: true and metrics.pixelBytes.

Types

export type VisionMode = boolean | 'only';

export interface VisionOption {
  vision?: VisionMode;
}

export interface AssertOptions extends VisionOption {
  timeout?: number;
  screenshot?: boolean;
  agent?: string;
}

export interface WaitForOptions extends VisionOption {
  timeout?: number;
  interval?: number;
  maxModelCalls?: number;
  agent?: string;
}

export interface ExtractOptions<Schema extends StandardSchemaV1> extends VisionOption {
  schema: Schema;
  timeout?: number;
  agent?: string;
}

export interface ActOptions {
  params?: AgentParams;
  timeout?: number;
  maxSteps?: number;
  maxModelCalls?: number;
  agent?: string;
}

export interface ActResult {
  readonly summary: string;
  readonly cache?: StepCacheInfo;
  readonly modelCalls: number;
  readonly actions: number;
}

export type AgentParam =
  | JsonPrimitive
  | Secret
  | readonly AgentParam[]
  | { readonly [key: string]: AgentParam };
export type AgentParams = Readonly<Record<string, AgentParam>>;

AgentError

export class AgentError extends Error {
  readonly code: AgentErrorCode;
  readonly explanation: string;
  readonly screenshot?: string;
  readonly blocked: boolean;
}

Property	Type	Description
`code`	`AgentErrorCode`	Closed union, assigned by the runner, never taken from model text
`explanation`	`string`	Sanitized, bounded prose. Same value as `message`.
`screenshot`	`string \| undefined`	Artifact path, present only when evidence was captured and permitted
`blocked`	`boolean`	True when the failure reports a blocked step (credentials, environment, or budget), not a product failure

Check code with isAgentError(error), exported from @e2edev/e2e, rather than instanceof, which does not hold across the runner’s module realms. Every code, its class, and its exit code are in Errors.

Usage accounting

Model usage is accounted per step and per run. A step reports tokenAccounting: 'provider' only when every call supplied complete provider token counts; otherwise the aggregate stays at adapter-upper-bound. Reported costs accumulate where the provider gives one. Token sums saturate at Number.MAX_SAFE_INTEGER and an overflowing cost total is omitted.

Test API

Project

Web

Mobile

Engine contract

Budgets

agent.act

Pixel tools: screenshot and tap_at

agent.assert

agent.waitFor

agent.extract

Vision

Types

AgentError

Usage accounting

How agent steps work

Agents and personas

Config

Errors

​Budgets

​agent.act

​Pixel tools: screenshot and tap_at

​agent.assert

​agent.waitFor

​agent.extract

​Vision

​Types

​AgentError

​Usage accounting

How agent steps work

Agents and personas

Config

Errors

Budgets

agent.act

Pixel tools: screenshot and tap_at

agent.assert

agent.waitFor

agent.extract

Vision

Types

AgentError

Usage accounting