Continuous integration

With command in your config, the job is install, browser, run.

.github/workflows/e2e.yml

name: e2e

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0
      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version: 26
          cache: pnpm
      - run: pnpm install --frozen-lockfile

      # Provision the browser as its own step, so its cost is visible in the log.
      - name: Install browsers
        run: npx playwright install chromium --with-deps

      # list keeps the log readable; junit writes .e2e/junit.xml for the CI test summary.
      - name: Run e2e
        run: npx --no-install e2e run --reporter list,junit
        env:
          APP_URL: http://127.0.0.1:3000
          E2E_USER_ADMIN_USERNAME: ${{ secrets.E2E_USER_ADMIN_USERNAME }}
          E2E_USER_ADMIN_PASSWORD: ${{ secrets.E2E_USER_ADMIN_PASSWORD }}

      # The reports exist on every outcome, so upload them whenever the job ran.
      - name: Upload report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: e2e-report
          path: |
            .e2e/report.json
            .e2e/junit.xml
          if-no-files-found: warn

      # Screenshots and traces are only interesting when something broke.
      - name: Upload artifacts
        if: failure()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: e2e-artifacts
          path: .e2e/artifacts
          if-no-files-found: warn
          retention-days: 7

Actions are pinned to commit SHAs. A tag is mutable, and a workflow with repository secrets is the wrong place to run code that can change under you. --no-install makes npx fail rather than fetch a mutable version from the registry; with --frozen-lockfile, the runner, the engine, and the browser all come from your lockfile. Point your CI’s test summary at .e2e/junit.xml and failures show up as test results instead of a red step. A startup failure such as APP_UNREACHABLE appears there too, as a run suite. For the failure itself on the pull request, add the GitHub reporter.

What CI changes

CI mode is on when the CI variable is set and is not empty, 0, or false. That is the only signal, so CI=1 npx --no-install e2e run reproduces CI behavior locally.

Setting	Local	CI	Why
`retries`	`0`	`1`	Locally a retry hides the bug you were about to inspect
`workers`	half your cores	`1`	A shared runner is the noisiest place to add timing variance
`cache`	`read-write`	`read-only`	A committed cache is untrusted input; see Caching agent steps
`.only`	allowed	rejected with `ONLY_IN_CI`	A focused test that reaches CI silently disables the rest of the suite
`command.reuseExisting`	honored	ignored, with a notice	A server already listening on a runner is a leftover, not your dev server

Override any of them when you mean it:

npx --no-install e2e run --workers 4 --retries 0

Telemetry attributes CI runs to the vendor rather than to a machine and prints no notice. Add E2E_TELEMETRY_DISABLED: '1' to the job’s env to send nothing from CI.

Model credentials

Deterministic tests need no model. For a suite that uses agent, the job passes the key the provider package reads:

      - name: Run agentic suite
        run: npx --no-install e2e run --config e2e.agent.config.ts
        env:
          AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}

The runner reads none of these variables, so the value never enters logs, reports, or the app process. Which variable each provider reads is on Choosing a model.

Keep model-backed tests opt-in

Deterministic tests are reproducible. Agentic tests can take a different, equally valid path on each run. Gating merges on them makes the model a shared dependency of every pull request.

Split the config

A separate e2e.agent.config.ts with its own tests glob and a larger timeout, since every planned step includes model round trips.

Split the workflow

Run it on workflow_dispatch or schedule, never on pull_request.

Keep it out of required checks

Branch protection requires the deterministic job only.

Nightly agentic run

on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:

Exit codes

A non-zero exit fails the step, which is usually all you want. When you react to the class of failure: 1 is a test failure, 2 is configuration or policy, 3 is infrastructure, 130 is a cancellation.

Do not retry the whole job on exit code 2. Configuration and collection errors are deterministic, so a re-run burns minutes and produces the same red. Exit 3 is the only code where a job-level retry is a reasonable reflex.

Pull request comments

One comment per run, edited in place on a rerun.

CLI reference

Flags, reporters, and every exit code.

Get started

Web

Mobile

Guides

Extend

Continuous integration

What CI changes

Model credentials

Keep model-backed tests opt-in

Exit codes

Pull request comments

CLI reference

​What CI changes

​Model credentials

​Keep model-backed tests opt-in

​Exit codes

Pull request comments

CLI reference

What CI changes

Model credentials

Keep model-backed tests opt-in

Exit codes