Writing tests

A test mixes two kinds of calls. Agent calls state a goal or ask a question. Deterministic calls pin down the parts that must not drift.

tests/todos.e2e.ts

import { test, expect } from '@e2edev/e2e';

test('todos survive a filter round-trip', async ({ app, agent, screen }) => {
  await app.open('/todos');

  await agent.act('add todos "Buy milk" and "Walk the dog", complete the first');
  await agent.assert('only one todo remains open');

  await screen.getByRole('tab', { name: 'Done' }).tap();      // pinned step
  await expect(screen.getByTestId('todo')).toHaveCount(1);    // pinned check
});

Goals: `agent.act`

Hand the model one goal. It plans and executes the steps under a deadline and a model-call budget, then your next line runs.

await agent.act('complete checkout with the test card');
await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });

Real values travel through params, never from the model’s imagination. A Secret param stays host-side: the model sees only its name, and the runner fills the field itself. See Signing in.

const member = credentials.user('member');
await agent.act('Sign in', {
  params: { username: member.username, password: member.password },
});

Write instructions as you would say them out loudName what is visible and prefer the exact wording on screen. One goal per call. The flow inside a goal is the model’s job; the order of goals is yours.

A step ends passed, failed, or blocked. A block means the step could not reach a product verdict. Configuration blocks such as missing credentials exit 2, and infrastructure blocks such as an unreachable environment exit 3. Budget exhaustion, timeouts, and unsupported automation still count as test failures and exit 1. How agent steps work lists the codes.

Checks: `assert`, `waitFor`, `extract`

assert asks one yes/no question about the current screen. A false answer is an ASSERTION_FAILED with the model’s explanation and a screenshot in the report.

await agent.assert('the dashboard shows a trial badge');

assert looks once. To wait for something to become true, poll with waitFor. In tree-only mode, unchanged observations skip the next judgment. Vision polling judges every interval because pixels can change while the tree stays the same.

await agent.waitFor('the export finishes and a download link appears', {
  interval: 500,
  timeout: 120_000,
});

extract pulls structured data off the screen so you can branch in TypeScript. Any Standard Schema validator works, zod included. Invalid output gets one repair attempt, then MODEL_OUTPUT_INVALID.

import { z } from 'zod';

const data = await agent.extract('every todo title and how many remain', {
  schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }),
});
expect(data.titles).toContain('Buy milk');

Assert on meaning, not on wordingDifferent models phrase things differently. expect(x.plan).toContain('Pro') survives a model swap; toBe('Pro plan') does not.

Judging pixels

The model reads a redacted text snapshot of the screen: roles, names, text, states. Add vision: true when the answer lives in pixels the snapshot cannot describe, such as a chart or a layout, or vision: 'only' to judge the screenshot alone.

await agent.assert('the chart trends upward', { vision: true });
await agent.assert('the search form is not covered by a banner', { vision: 'only' });

Images cost input tokens on every call. The agent reference lists every mode.

Deterministic APIs: `screen` and `expect`

When a step or a check is too important to leave to a model, write it exactly. These calls cost no tokens and never vary.

// Semantic queries: role and name first, label second, test ID last.
await screen.getByRole('button', { name: 'Sign in' }).tap();
await screen.getByLabel('Email').fill('ada@example.test');
await screen.getByTestId('todo').first().tap();

// Scope into a row, gesture like a user:
const row = screen.getByRole('listitem').filter({ hasText: 'Design review' });
await row.getByRole('button', { name: 'Archive' }).tap();
await screen.swipe({ direction: 'up', momentum: 'fast' });

// Polling assertions:
await expect(screen.getByRole('status')).toHaveText('2 remaining');
await expect(web).toHaveURL('/dashboard');

Two rules keep these honest. A query matching two nodes is an error, not a first-match guess; narrow with .filter(), .first(), or .nth(). Locator assertions retry while direct reads like textContent() do not, so assert with the matcher. Browser-only power lives on the web fixture: routes, cookies, dialogs, frames, downloads. Import test from @e2edev/playwright to have it typed. See the Playwright reference and the screen reference. A resource every test needs, a seeded workspace or a signed-in account, is a fixture of your own: test.extend({ workspace: async ({ web }, use) => { ... } }) returns a test that sets it up before each body and tears it down after, failed or not. See Your own fixtures.

Structure

test.describe('billing', { tags: ['billing'] }, () => {
  test.beforeEach(async ({ app }) => { await app.open('/billing'); });

  test('upgrades', { retries: 2, timeout: 60_000 }, async ({ agent }) => {
    await agent.act('upgrade to Pro');
  });
});

test.skip('not ready yet', async () => {});
test.only('just this one while I work', async () => {}); // rejected in CI

Common options: timeout (default 120 s), retries, tags (filter with --tag), session (state from a setup test), agentContext (extra context for this test’s agent calls), and agent (which configured agent runs the test). Group options apply to their tests. The test reference has every rule. Tests are independent and start from clean state. When a flow must span several tests, mark the group { serial: true }: members share one app state, run in order on one worker, and retry as a whole.

Signing in

Sessions and secrets.

Agents and personas

Several agents in one suite, and running a flow as each of them.

Get started

Web

Mobile

Guides

Extend

Writing tests

Goals: `agent.act`

Checks: `assert`, `waitFor`, `extract`

Judging pixels

Deterministic APIs: `screen` and `expect`

Structure

Next

Signing in

Agents and personas

​Goals: agent.act

​Checks: assert, waitFor, extract

​Judging pixels

​Deterministic APIs: screen and expect

​Structure

​Next

Signing in

Agents and personas

Goals: `agent.act`

Checks: `assert`, `waitFor`, `extract`

Judging pixels

Deterministic APIs: `screen` and `expect`

Structure

Next