Email Testing in CI: Run Verification Tests Reliably

Email verification is usually the flakiest step in a CI pipeline, and almost always for the same two reasons: a shared inbox that parallel shards fight over, and a timeout tuned to a fast laptop that a loaded CI runner blows past. Fix those two and the test that failed one build in five becomes as boring as any other assertion. This post is the CI-specific playbook — how to pick an inbox architecture, isolate per shard, handle the API key, budget timeouts against real mail latency, and decide whether an email test blocks a merge.

The framework tutorials live elsewhere: the Playwright email testing guide and Cypress email testing guide each walk through a full spec, and the Selenium email verification guide covers the WebDriver flavor. This page sits above them — the pipeline concerns that are the same no matter which runner drives the browser.

Quick answer

Email testing in CI works reliably when you follow four rules. First, use a real disposable inbox per test — not a shared mailbox, not a mock — so parallel shards never cross wires. Second, keep the API key in your CI secret store and expose it to the job as an environment variable, never in the repo. Third, set the poll deadline above your real p99 mail latency and the job-level timeout above that, so a slow delivery fails cleanly instead of hanging the pipeline. Fourth, quarantine new email tests in a non-blocking job until their pass rate earns them a place in the required checks.

The rest is detail: which of the three inbox architectures to pick, how per-shard isolation actually works, and where the rate limit bites when you fan out.

The three architectures, and which one survives CI

Every approach to email testing is one of three shapes. They are not equal, and the trade-off sharpens under CI's parallelism.

Real inbox over IMAP — real, but flaky. You point tests at a Gmail or Outlook account and read it over IMAP. This tests genuine delivery, but it drags a mail-client integration into your suite: OAuth or app passwords, provider rate limits, and a single account whose state bleeds between runs. In CI, where jobs run concurrently, that shared account is a race waiting to happen — worker A reads worker B's code.

SMTP capture like GreenMail or Mailtrap — fast, but not real delivery. You run a fake SMTP server (GreenMail in-process, Mailtrap as a hosted sink) that catches whatever your app sends, then read it back over an API. This is fast and deterministic, and a fine choice when what you want to assert is "did my app try to send the right email." What it does not test is delivery: the message never leaves your own infrastructure, so DNS, the real sending path, and anything a downstream provider does to the mail sit outside the test. Green here means the template rendered, not that a user would receive it.

Disposable-inbox API — real delivery without shared state. You call an API to mint a throwaway inbox per test, use its address in the signup, and read the delivered message back over HTTP. The mail travels the real path — your app sends it, it gets delivered, your test reads what landed — but each inbox is unique and discarded, so there is no shared state to collide on. This is the architecture that holds up under CI parallelism, and it is what the rest of this post assumes. The framework-agnostic argument for it lives in the temp mail API automation guide.

Most pipelines want both: SMTP capture for fast "did we send it" checks, and a disposable-inbox API for the few true end-to-end flows that need to prove real delivery. Pick per assertion, not per project.

Per-shard inbox isolation

Parallelism is why CI email tests fail when local ones pass. Your CI provider splits the suite across shards and runs them at once. If two shards share an inbox, they read each other's mail, and the failure is maddening: green locally where tests run one at a time, red in CI, never the same test twice.

The fix is structural, not a retry. Mint the inbox in the per-test setup hook so every test gets its own address:

// One inbox per test — runs in beforeEach, so every parallel worker
// and every shard gets a unique address with no coordination.
let inbox;

beforeEach(async () => {
  inbox = await createInbox(); // POST https://tempmaily.co/api/v1/inboxes
});

afterEach(async () => {
  if (inbox) await deleteInbox(inbox.id); // 404 in teardown is fine
});

Because the address is unique per test, sharding is free — split across 2 shards or 20 and no test can see another's mail, with no shared counter or lock to coordinate. The createInbox and deleteInbox helpers are the ones the Playwright and Cypress guides build; here the point is only where they run — per test, so isolation falls out of the structure.

The anti-pattern is hoisting inbox creation to a suite-level beforeAll to "save API calls." That hands one address to many tests and reintroduces the cross-shard race disposable inboxes exist to kill. Save calls by not creating an inbox per assertion (below), not by sharing one across tests.

Secrets: the API key belongs in the CI secret store

The TempMaily API key is a Bearer credential — anyone holding it can create and read inboxes on your account. It never goes in the repo. Store it in your CI provider's secret store and expose it to the job as an environment variable the test reads from process.env.XTM_KEY.

In GitHub Actions that is a repository secret under Settings, Secrets and variables, Actions. In GitLab it is a CI/CD variable, set to masked and protected. Both keep the value out of source control and out of log output. The one thing to avoid is echoing it — never run: echo $XTM_KEY for debugging, because CI logs are often readable by more people than the secret store is.

For a monorepo that fans out many parallel jobs, consider a dedicated CI key separate from the one developers use locally. That gives CI its own rate-limit budget and lets you rotate the CI credential without disrupting local work.

Timeout budgets: two clocks, set in the right order

The single most common CI email flake after shared inboxes is a timeout mismatch. There are two clocks and they must be ordered.

The poll deadline is your own loop's patience — how long waitForMessage keeps checking the messages endpoint before giving up. The job timeout is the pipeline's patience for the whole test or step. Real verification mail usually lands in a second or two, but a cold sender, a queued send job, or a busy CI runner can push it past ten. Set the poll deadline above your measured p99 delivery time, not at it — 30 seconds is a comfortable default for mail that normally arrives in two.

Then set the job timeout above the poll deadline. If your poll waits 30 seconds but the CI step's own timeout is shorter, the runner kills the job mid-poll and you get an opaque "job exceeded time limit" instead of a clean "no matching message arrived." Order them poll-deadline-below-job-timeout and the loop always owns its failure — the error you can actually read.

// Poll deadline comfortably above p99 mail latency. The job/step timeout
// in your CI config must sit ABOVE this so the runner never kills the
// poll mid-loop and hands you an unreadable timeout instead.
const message = await waitForMessage(inbox.id, (s) => /verify|confirm|code/i.test(s), {
  timeoutMs: 30_000,
  intervalMs: 2_000,
});

A complete GitHub Actions workflow

Here is a working workflow that runs a Playwright suite with the TempMaily API, sharded across two runners. The moving parts specific to email testing are the XTM_KEY secret, the shard matrix, and the step timeout sitting above the in-test poll deadline.

# .github/workflows/e2e.yml
name: e2e
on: [push]

jobs:
  playwright:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run email verification tests
        timeout-minutes: 10 # above the 30s in-test poll deadline
        run: npx playwright test --shard=${{ matrix.shard }}/2
        env:
          XTM_KEY: ${{ secrets.XTM_KEY }}

A few things earn their place. fail-fast: false lets both shards finish even if one fails, so you see every failure in one run. Every shard gets XTM_KEY from the same secret — that is fine, because inbox isolation happens per test, not per key. And timeout-minutes: 10 is the job clock sitting well above the 30-second poll, so a genuinely stuck delivery surfaces as an assertion failure long before the step is killed.

GitLab CI and retries

The same shape ports to GitLab — secrets are CI/CD variables, parallelism is the parallel keyword, and retry handles the rare infra blip:

# .gitlab-ci.yml
e2e:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 2
  timeout: 10 minutes
  retry:
    max: 1
    when: runner_system_failure
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  # XTM_KEY is a masked, protected CI/CD variable — not in this file.

Note the retry is scoped to runner_system_failure, not to test failures. Retrying a failed email test at the job level is a blunt instrument — it re-runs the whole suite to paper over one flaky test, and it hides the flake instead of fixing it. Prefer your test runner's own retry (Playwright's retries, Cypress's retries: { runMode: n }), which re-runs just the failed test and re-runs its beforeEach, so a retried attempt gets a brand-new inbox with no message bleed from the failed attempt. Job-level retry should catch infrastructure, not mask product flake.

Quarantine before you block merges

A new email test has not earned the right to block your team's merges on day one. Run it, but do not gate on it yet.

The pattern is a non-blocking job. Put new or historically flaky email tests in a separate CI job that runs on every push but is not a required status check, so a failure is visible without stopping the merge. Watch its pass rate for a week or two. Once it is consistently green — say 20 clean runs in a row — promote it into the required check that gates the merge. That earns you the signal to trust the test without a genuine flake holding the team hostage while you tune it.

The reverse also matters: when a previously-trusted test starts flaking, demote it back to the quarantine job while you investigate. A required check that is red half the time trains everyone to click merge anyway.

Cost and rate-limit hygiene

The TempMaily key is capped at 300 requests per 60 seconds, and 429 rate_limited is the response when you go over. A single test is nowhere near that ceiling — one inbox creation, a handful of 2-second polls, one delete. The ceiling comes into view when many parallel shards poll at once, so a little discipline keeps you clear.

One inbox per test, not per assertion. Mint the inbox once in setup and reuse it for every poll and assertion in that test. Creating a fresh inbox per assertion multiplies your API calls for no benefit and pushes you toward the limit. A suite of 40 email tests should make roughly 40 creation calls, not 400.

Keep the poll interval in seconds, not milliseconds. A 2-second interval is about 30 requests a minute per test — well within budget. A tight while loop with no delay is both wasteful and a fast route to 429.

Give CI its own key if you fan out heavily. A dozen shards each polling several inboxes add up, so a dedicated CI key keeps that budget separate and stops one heavy pipeline from rate-limiting a developer mid-debug. Delete inboxes in teardown too — it frees your account's storage quota.

Common mistakes

• Sharing one inbox across tests or shards. Works locally, collapses the first time CI runs two shards at once. Mint one inbox per test in the setup hook.
• Poll deadline below job timeout, reversed. If the CI step timeout is shorter than your in-test poll, the runner kills the job mid-poll and you lose the real error. Job timeout above poll deadline, always.
• Committing the API key or echoing it in logs. It is a Bearer credential. Keep it in the secret store, mask it, and never echo it for debugging.
• Creating an inbox per assertion. Multiplies API calls and marches you toward the 300-per-60s rate limit. One inbox per test, reused for every check inside it.
• Job-level retry to hide flake. Retrying the whole suite masks a broken test. Use the framework's per-test retry (which also gives a fresh inbox), and reserve job retry for infrastructure failures.
• Blocking merges on a brand-new email test. Quarantine it in a non-blocking job until its pass rate earns promotion, or one flaky test stalls the whole team.
• Matching any six digits. \b\d{6}\b matches a copyright year or an order number before the real code. Anchor the regex to nearby copy like "your code is", and read bodyText (falling back to bodyHtml), not text/html.
• Testing with a domain your app blocklists. If your product rejects known disposable domains, a free address bounces off your own validation before the flow starts. Use a Premium dedicated or custom domain the app treats as legitimate, as covered in test email addresses for QA.

Next steps

Reliable email testing in CI is mostly about removing shared state and ordering your clocks. Give every test its own real inbox so shards never collide, keep the API key in the secret store, set the poll deadline above real mail latency and the job timeout above that, quarantine new tests until they earn a required check, and reuse one inbox per test to stay under the rate limit. Do that and the email step stops being the thing everyone blames for red builds.

For the runner-specific code, start from the Playwright, Cypress, or Selenium guide; for the framework-agnostic poll-and-extract pattern and the webhook alternative, the temp mail API automation guide has it. Programmatic access is part of TempMaily Premium, and the API reference is the source of truth for every endpoint, field, and limit used above.