> ## Documentation Index > Fetch the complete documentation index at: https://docs.e2e.army/llms.txt > Use this file to discover all available pages before exploring further. # agent > Planned flows, judgments, extraction, and their budgets. Every `agent` call is one bounded invocation: a fresh observation, an explicit deadline, a model-call budget, and no shared transcript between calls. Acquiring the fixture without a configured model fails with `MODEL_UNAVAILABLE`. ```ts theme={"theme":"catppuccin-mocha"} export interface Agent { act(instruction: string, options?: ActOptions): Promise; waitFor(condition: string, options?: WaitForOptions): Promise; extract( instruction: string, options: ExtractOptions, ): Promise>; assert(assertion: string, options?: AssertOptions): Promise; } ``` ## Budgets Model calls count every model request including repair rounds. Action steps count committed engine actions. Every timeout is additionally capped by the remaining test timeout. | Method | Model calls | Action steps | Default timeout | | --------- | --------------------------------: | ---------------------------: | --------------------------- | | `act` | `agents..maxModelCalls`, 25 | `agents..maxSteps`, 25 | `config.timeout`, 120000 ms | | `waitFor` | `agents..maxModelCalls`, 25 | 0 | the judgment budget | | `extract` | 2 | 0 | the judgment budget | | `assert` | 2 | 0 | the judgment budget | The judgment budget is `max(30000, config.actionTimeout)` milliseconds. Per-call budgets must be positive integers and can only lower the configured limits. Every duration option is milliseconds. The `assert` row holds for the built-in agent. When the selected agent has a custom `executor`, `assert` is dispatched to it as a step: its budget is the agent's `maxSteps` and `maxModelCalls`, its default timeout is `config.timeout`, and `vision` or `screenshot` on the call is `UNSUPPORTED_CAPABILITY`. ## agent.act ```ts theme={"theme":"catppuccin-mocha"} act(instruction: string, options?: ActOptions): Promise; ``` Plans and executes one bounded multi-action flow. A passing verdict resolves with an `ActResult`; a failing or blocked one throws an `AgentError` whose `code` and `blocked` flag say why. | Name | Type | Default | Valid | | ----------------------- | ------------- | --------------------------- | ----------------------------------------------------------------------- | | `instruction` | `string` | | 1 through 8192 UTF-8 bytes after NFC | | `options.params` | `AgentParams` | `undefined` | JSON-safe values plus `Secret`, at most 64 KiB and 32 levels deep | | `options.timeout` | `number` | `config.timeout` | positive safe integer | | `options.maxSteps` | `number` | the agent's `maxSteps` | positive integer, at most the resolved limit | | `options.maxModelCalls` | `number` | the agent's `maxModelCalls` | positive integer, at most the resolved limit | | `options.agent` | `string` | the test's agent | a name in `agents`; unknown is `INVALID_ARGUMENT` before the step opens | Values come from your code through `options.params`, never from the model. A `Secret` in `params` is projected before the model sees anything: the model receives the secret's name and purpose, and the harness fills the value host-side under the same authorization as `locator.fill`: the credential must be registered, the current origin allowed by both the target and the credential, and the field an enabled secure input whose purpose matches. ```ts theme={"theme":"catppuccin-mocha"} await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); const member = credentials.user('member'); const { summary, cache } = await agent.act('Sign in', { params: { username: member.username, password: member.password }, }); ``` The result carries the step's `summary`, the `modelCalls` and `actions` it spent, and `cache`: how the [trace cache](/cache) took part (`self-finalized`, `agent-concluded`, or `missed` with a `reason`; absent when caching is off). ### Pixel tools: screenshot and tap\_at `act` takes no `vision` option. Pixels reach an `act` step through two tools the built-in agent offers while pixels can leave the runner. The act model itself receives the image: no separate vision model, no extra model call, and nothing lost in a description. | Tool | What it does | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `screenshot()` | Attaches a masked screenshot of the viewport to the tool result, with the changes since the screen the model holds. From then on the step is in pixel mode: every action result carries a fresh screenshot, so the model sees what each action did. | | `tap_at({ x, y })` | Taps a point given in the latest screenshot's pixel coordinates. The harness scales the point into the observation's CSS pixels and hit-tests it against the tree: on an engine with `perform`, a listed control under it is tapped by its id, exactly like `tap`, with a replayable descriptor; a point on nothing listed, or any point on an engine that taps by point only, is tapped as a bare point through the engine's `tapAt`. The result names what was tapped or what sits under the point, and carries a fresh screenshot. | While the step is showing pixels, the wait after an action watches the pixels too: a tap that redraws a canvas is read as soon as the redraw lands, instead of waiting out the change window on a tree that never moves. A screen that lists nothing to act on by id (a canvas, a game, a semantics-free native screen) opens with a screenshot already attached, so the model never spends a turn discovering that it needs one; any screen with a listed control opens tree-only and the model asks when the tree falls short. Older screenshots are elided from the conversation in batches, keeping the newest two, so a long flow in pixel mode carries a bounded number of images, and each is resampled to a long side of at most 768 pixels before it goes to the model: on a drawn keypad and a drawn wizard that size kept every run passing at a third less cost than the full capture, and 512 did not. Coordinates the model returns are in that image; the harness scales them back. Both tools are offered while no secret has been filled when the step opens; a secret filled during the step keeps them listed but withholds every later screenshot (`PIXEL_TAINTED`), and each result says so; `tap_at` needs an engine that can tap by node (`perform`) or by point (`tapAt`, the `pointer` capability, which the Playwright and agent-device engines declare), and on an engine without `tapAt` the point must land on a listed control. A bare-point tap replays from the [trace cache](/cache) the way a coordinate-driven tool does: the same point on a viewport of the recorded size, or, when a listed node contained the point, the same place inside that node's live box. A viewport of another size or a node that cannot be re-found hands the step to the model, and the recorded end state still has to return before a replay passes on its own. A step that sent pixels records `visionInput: true` and `metrics.pixelBytes`. The pixel path asks the act model to ground a point in an image, and models differ widely there. Measured on a drawn ten-key keypad and an eight-screen drawn wizard: the gpt-5.6 luna family and Gemini 3.8 Flash completed both; Claude Sonnet 5 completed the wizard's large shapes but never the keypad's small keys; DeepSeek and GLM flash models gave up, and their providers bill a 1280 by 720 screenshot at tens of thousands of input tokens against a few hundred on OpenAI. Pin an act model that grounds well when your flows leave the tree. Throws `ACTION_FAILED` for a failed verdict with no more specific code; a blocked code (`AUTH_CREDENTIAL_UNAVAILABLE`, `ENVIRONMENT_UNAVAILABLE`, `SEED_DATA_MISSING`, `TEST_SETUP_FAILED`, `AUTOMATION_UNSUPPORTED`) when the step could not reach a product verdict; `STEP_BUDGET_EXHAUSTED` and `STEP_TIMEOUT` when a budget or the deadline ran out; `INVALID_ARGUMENT` for a malformed instruction, oversized `params`, or an unknown option; and `UNSUPPORTED_CAPABILITY` for `schema` or `vision`, which `act` does not take. ## agent.assert ```ts theme={"theme":"catppuccin-mocha"} assert(assertion: string, options?: AssertOptions): Promise; ``` One fresh observation, exactly one judgment. It does not poll. | Name | Type | Default | | -------------------- | ------------ | ---------------------------------- | | `assertion` | `string` | | | `options.timeout` | `number` | `max(30000, config.actionTimeout)` | | `options.screenshot` | `boolean` | `true` | | `options.vision` | `VisionMode` | the agent's `vision`, `false` | | `options.agent` | `string` | the test's agent | Throws `ASSERTION_FAILED` on a false judgment, carrying the explanation and, when permitted, a screenshot path. Evidence is skipped with `screenshot: false` and once any secret has been filled in the attempt. `screenshot` controls failure evidence attached after the judgment; `vision` controls what the model is shown before it. ```ts theme={"theme":"catppuccin-mocha"} await agent.assert('the dashboard shows a trial badge'); ``` ## agent.waitFor ```ts theme={"theme":"catppuccin-mocha"} waitFor(condition: string, options?: WaitForOptions): Promise; ``` One observation per interval, and a judgment when the screen changed. Succeeds on the first true judgment. | Name | Type | Default | Valid | | ----------------------- | ------------ | ---------------------------------- | -------------------------------------------- | | `condition` | `string` | | | | `options.timeout` | `number` | `max(30000, config.actionTimeout)` | positive safe integer | | `options.interval` | `number` | `3000` | integer 100-60000, milliseconds | | `options.maxModelCalls` | `number` | the agent's `maxModelCalls` | positive integer, at most the resolved limit | | `options.vision` | `VisionMode` | the agent's `vision`, `false` | | | `options.agent` | `string` | the test's agent | a name in `agents` | Throws `STEP_TIMEOUT` on deadline expiry and `STEP_BUDGET_EXHAUSTED` on budget exhaustion, each carrying the last judgment, and `INVALID_ARGUMENT` for an out-of-range option or one `waitFor` does not take. ```ts theme={"theme":"catppuccin-mocha"} await agent.waitFor('the upload progress reports completion', { interval: 500 }); ``` ## agent.extract ```ts theme={"theme":"catppuccin-mocha"} extract( instruction: string, options: ExtractOptions, ): Promise>; ``` Takes one fresh observation, requests structured output, validates it, and returns the validated value. It performs no app actions. | Name | Type | Default | Valid | | ----------------- | ------------ | ---------------------------------- | -------------------------------- | | `instruction` | `string` | | | | `options.schema` | `Schema` | | any Standard Schema v1 validator | | `options.timeout` | `number` | `max(30000, config.actionTimeout)` | positive safe integer | | `options.vision` | `VisionMode` | the agent's `vision`, `false` | | | `options.agent` | `string` | the test's agent | a name in `agents` | The budget is two model calls: one extraction and one repair round for output that failed validation. Throws `INVALID_ARGUMENT` for a schema that is not Standard Schema v1, and `MODEL_OUTPUT_INVALID` when validation still fails after the repair. ```ts theme={"theme":"catppuccin-mocha"} const summary = await agent.extract('the workspace name and plan', { schema: z.object({ name: z.string(), plan: z.string() }), }); ``` ## Vision `assert`, `waitFor`, and `extract` accept `vision`. It controls what evidence the model is shown. Sending pixels needs a model that accepts images; images are billed as input tokens. `act` has no such option; its pixel path is the `screenshot` and `tap_at` tools above. | Mode | What the model sees | Cost | | -------- | -------------------------------------------- | ----------------------------------------- | | `false` | the tree | no image tokens | | `true` | the tree and a masked screenshot, every call | image tokens on every call | | `'only'` | the screenshot, and not the tree | image tokens, minus the observation bytes | Set `vision` on the agent in config to make a mode the project default; a per-call value wins. Pixels go to the agent's `model`, the same one the act loop shows its screenshots to. **Use `'only'` for questions about what the page presents** `vision: true` sends the tree *and* the screenshot, and the tree is a cheaper path to an answer. A consent sheet covering a form, a price at `opacity: 0`, a button under a cookie banner: the tree reports all of them as fine. ```ts theme={"theme":"catppuccin-mocha"} // Passes on the tree alone while a consent sheet covers the form. await agent.assert('the search form is visible and not covered', { vision: true }); // Judged on the screenshot. There is no tree in the request to answer from. await agent.assert('the search form is visible and not covered', { vision: 'only' }); ``` Use `true` when the tree is part of the answer, such as a disabled state that is not visible at all. The screenshot is the viewport, never the full page, so `'only'` cannot see below the fold. Write `'only'` conditions about what is on screen and let the flow scroll first. Pixel evidence degrades instead of failing the call in every mode but `'only'`. It is withheld, and the tree sent alone, when a secret fill has tainted the viewport (`visionDegraded: "PIXEL_TAINTED"` on the step), when the engine masked fewer regions than the secure fields it observed (`MASKING_UNPROVEN`), or when the engine cannot capture pixels (`UNSUPPORTED_CAPABILITY`). `'only'` fails with `POLICY_DENIED` in those cases rather than answer a question about pixels from the tree. A step that sent pixels records `visionInput: true` and `metrics.pixelBytes`. ## Types ```ts theme={"theme":"catppuccin-mocha"} export type VisionMode = boolean | 'only'; export interface VisionOption { vision?: VisionMode; } export interface AssertOptions extends VisionOption { timeout?: number; screenshot?: boolean; agent?: string; } export interface WaitForOptions extends VisionOption { timeout?: number; interval?: number; maxModelCalls?: number; agent?: string; } export interface ExtractOptions extends VisionOption { schema: Schema; timeout?: number; agent?: string; } export interface ActOptions { params?: AgentParams; timeout?: number; maxSteps?: number; maxModelCalls?: number; agent?: string; } export interface ActResult { readonly summary: string; readonly cache?: StepCacheInfo; readonly modelCalls: number; readonly actions: number; } export type AgentParam = | JsonPrimitive | Secret | readonly AgentParam[] | { readonly [key: string]: AgentParam }; export type AgentParams = Readonly>; ``` ## AgentError ```ts theme={"theme":"catppuccin-mocha"} export class AgentError extends Error { readonly code: AgentErrorCode; readonly explanation: string; readonly screenshot?: string; readonly blocked: boolean; } ``` | Property | Type | Description | | ------------- | --------------------- | --------------------------------------------------------------------------------------------------------- | | `code` | `AgentErrorCode` | Closed union, assigned by the runner, never taken from model text | | `explanation` | `string` | Sanitized, bounded prose. Same value as `message`. | | `screenshot` | `string \| undefined` | Artifact path, present only when evidence was captured and permitted | | `blocked` | `boolean` | True when the failure reports a blocked step (credentials, environment, or budget), not a product failure | Check `code` with `isAgentError(error)`, exported from `@e2edev/e2e`, rather than `instanceof`, which does not hold across the runner's module realms. Every code, its class, and its exit code are in [Errors](/reference/errors). ## Usage accounting Model usage is accounted per step and per run. A step reports `tokenAccounting: 'provider'` only when every call supplied complete provider token counts; otherwise the aggregate stays at `adapter-upper-bound`. Reported costs accumulate where the provider gives one. Token sums saturate at `Number.MAX_SAFE_INTEGER` and an overflowing cost total is omitted.