Playwright vs Bot Arena

Verdict: the canonical checks here are a single-line addInitScript away from passing — they are the textbook example a stealth tutorial opens with.

Each of the five remaining signals can in principle be spoofed from Playwright:

• navigator.webdriver can be hidden via --disable-blink-features=AutomationControlled plus an addInitScript that redefines the property.
• The User-Agent can be spoofed with --user-agent="..." to strip the HeadlessChrome token.
• navigator.plugins, navigator.languages, and the Notification.permission / permissions.query pair can all be patched via Object.defineProperty in an init script.

Off-the-shelf stealth bundles (playwright-extra + puppeteer-extra-plugin-stealth) ship most of these patches already. The catch: every Chrome release introduces new tells, and commercial bot-detection vendors (Cloudflare, DataDome, PerimeterX, Imperva) maintain fingerprint databases of every known stealth-plugin signature. You spend more time updating your evasions than writing tests, and you only ever win temporarily.

Verdict: partially patchable in script; the pixel measurements require effectively rebuilding what VNC-AIVA already is.

The JavaScript-level signals (chrome.app, chrome.csi, driver shims, toString integrity) can be polyfilled with an addInitScript at page load. Easy.

The window/screen pixel measurements are different. outerHeight - innerHeight = 0 is true because the headless browser literally has no toolbars. Two ways out, neither great:

1. Run headed (headless: false) on a server with Xvfb/Xvnc. But then you need a real desktop environment with a window manager and a panel to populate screen.availHeight < screen.height, plus you need Chrome to actually display its chrome (not --kiosk). At that point, you have rebuilt the AIVA architecture from scratch.
2. Spoof the values from JS — override window.outerHeight, screen.height, etc. via addInitScript. But the spoofs need to be internally consistent across signals: if you claim a 1080-pixel screen with a 40-pixel taskbar, the browser viewport's actual height needs to plausibly fit inside that. Cross-signal correlation catches these mismatches.

In practice: an automation team trying to fix L2 with Playwright ends up reinventing AIVA badly.

Verdict: bypassable for the basic checks the arena performs, but Bezier-curve plugins alone do not defeat Cloudflare Bot Management or DataDome's behavioural model in production.

Playwright does expose lower-level mouse APIs that can generate intermediate moves:

• page.mouse.move(x, y, { steps: 30 }) emits 30 intermediate mousemove events along a straight line.
• Wrap that in a Bezier-curve helper with randomized jitter and you produce trajectories with the right shape and curvature.
• page.keyboard.type(text, { delay: rand(80, 200) }) dispatches one key at a time with randomized inter-key delays.

The catch: every interaction in the test suite needs this treatment. A one-line page.click() becomes a thirty-line "humanize" helper. And advanced behavioural fingerprinting (used by serious bot-detection vendors) trains ML models on real human mouse telemetry — they pick up on acceleration curves, overshoot-and-correct patterns, pause-before-click latency, and dozens of other features that synthetic Bezier curves don't replicate. So: bypassable here, in this demo. Increasingly hard against production-grade defenders.

Verdict: solvable via stealth-class plugins, with continuous cat-and-mouse against sophisticated vendors.

Canvas / audio / WebGL renderer / font spoofing is the headline feature of puppeteer-extra-plugin-stealth, playwright-extra's stealth bundle, rebrowser-patches, and Camoufox. Drop one of these into a Playwright setup and the four signals this demo measures are routinely defeated; the maintainers have already solved cross-signal consistency for the common case — a "Chrome on Windows" fingerprint package is internally coherent across canvas hash, audio waveform, WebGL renderer, and font widths.

The friction (4/5, not impossible) lives in the cat-and-mouse:

• Detection vendors publish writeups identifying new tells in stealth packages and patch around them — see DataDome on stealth's iframe-contentWindow leak.
• Rebrowser-patches and Camoufox ship as continuously-updated drop-ins; staying current means tracking releases on the same cadence as the detection vendors.
• Higher-end defenders (Akamai) have moved primary detection to TLS-level fingerprinting (JA3/JA4), which neither stealth nor patched Playwright addresses on its own — that adds another tooling layer.

For the named vendors and most production-grade detection, a Playwright suite picks a stealth stack and accepts a perpetual maintenance overhead. Not impossible, not trivial.

Verdict: functionally impossible to bypass from inside Playwright. The only working "fix" is to outsource the problem.

Turnstile's logic is intentionally closed-source. It runs every kind of signal the previous four levels illustrate, plus a stack of private checks Cloudflare keeps to itself, plus IP reputation, plus behavioural analysis trained on the firehose of real human traffic across the Cloudflare network. Even a Playwright author who perfectly fixed levels 1-4, ran from a residential IP, and hand-rolled humanized interactions would still be classified as automated with high confidence — Cloudflare's behavioural model is too good.

The "solution" used in the wild is paid CAPTCHA-solver services (2Captcha, anti-captcha, CapMonster, etc.). They route the challenge through real-browser farms — either real humans or sophisticated stealth setups — and return a valid token in a few seconds, for a few cents each. Wire one of those into your test:

const token = await solver.solveTurnstile({
  sitekey: '0x4AAAAAADOBZMoei4aG9CNO',
  url: 'https://bot-arena.jhero.app/bot-detection/level-5/',
});
await page.evaluate((t) => {
  document.querySelector('input[name="cf-turnstile-response"]').value = t;
}, token);

This works — but it has defeated the original purpose of using Playwright. You have paid a third-party service to act as the human in front of the human-detector. Your "automated" tests now have a per-run cost and a human-in-the-loop dependency. This is exactly the kind of corner that VNC-AIVA, by being a real browser session at the OS level, avoids without any third-party dependency.

Verdict: structurally unreachable for Playwright when the canvas surface does not expose an accessibility tree — which is the default for legacy and enterprise apps, the apps automation actually targets.

Playwright's locator engine reads the DOM. Against a canvas-only surface there is nothing to read:

• page.getByLabel('Email') — no <label> element exists.
• page.getByRole('textbox') — no <input> element exists.
• page.getByText('Sign in') — the text "Sign in" is painted pixels, not a text node.

The blocking property is what makes this 5/5 rather than a softer 'workaround' number. A canvas-rendered surface anywhere in a workflow blocks the entire automation — selectors cannot skip past a step they cannot interact with. One signature-pad widget, one canvas-based chart that gates "continue", one canvas-rendered datepicker, and the whole test breaks. Effort per-canvas does not compose down to a small total when any single canvas means the run cannot complete.

The narrow exception: a handful of consumer-SaaS canvas apps ship a parallel accessibility-tree HTML layer for screen readers, which Playwright's getByRole / page.accessibility.snapshot() can reach. Figma and Google Sheets are the canonical examples. But Figma is rare in the apps an enterprise automation team actually targets. The bulk of canvas-in-the-wild — Photoshop Web, Photopea, tldraw, Excalidraw, Miro, Unity/Unreal WASM games, legacy AS/400-to-web emulators, custom signature pads, in-house dashboards built on D3-canvas — ship no accessibility tree at all.

The only theoretical Playwright path for the no-a11y case is to screenshot from Playwright, pipe to an external OCR / template-matching pipeline, and use page.mouse.click(x, y) at the resolved coordinates. At that point you have built a worse version of the classic AIVA — and you have moved the actual automation outside Playwright entirely.

Verdict: depends on whether the app ships accessibility metadata. With a11y, Playwright's recommended semantic locators (getByRole, getByLabel, getByText) are immune to id/class rerolling and the row is close to 2/5. Without a11y — the arena's demo deliberately strips it — only brittle structural selectors are left and it's closer to 4/5. 3/5 fits the average.

Real-world ground truth on the named examples:

• CSS-in-JS (Tailwind, styled-components, Emotion): Tailwind classes are stable utility strings; Emotion hashes are stable per style definition, not per request. getByRole + accessible name handles all of these cleanly.
• Anti-bot WAFs: DataDome, PerimeterX, Kasada rely on TLS+JS fingerprinting, behavioural biometrics, and proof-of-work — per-request DOM-selector randomisation is not a documented production tactic.
• Ticketing / sneaker drops: Ticketmaster does have some "moving selectors" but its primary defence is randomised queue position + fingerprinting; Nike SNKRS uses telemetry + IP reputation. Selector churn happens but is rarely the load-bearing defence.

For this demo (no labels, no accessible names, all attributes rerolled), the fallbacks are:

1. page.locator('input[type="email"]') — works this run; breaks if the input type is also randomised, or another email input is added.
2. page.locator('input').nth(0) — works this run; breaks the moment the form reorders or grows.
3. page.locator('div:has-text("Email") + input') — works for this layout; breaks if the DOM structure is rewritten.

For an app that combines randomised attributes with stripped accessibility metadata, every fallback is one revision away from breaking; the maintenance burden grows linearly with the number of forms.

Verdict: solvable in Playwright via a documented workaround, but with timing-sensitive caveats — and most real enterprise apps avoid closed shadow anyway.

The Playwright maintainers' own suggestion (issue #23047) is to monkey-patch Element.prototype.attachShadow in an addInitScript hook so every subsequent attachShadow({mode:'closed'}) call actually returns an open root:

await context.addInitScript(() => {
  const orig = Element.prototype.attachShadow;
  Element.prototype.attachShadow = function (options) {
    return orig.call(this, { ...options, mode: 'open' });
  };
});

Playwright's normal piercing locators then work as if the app had opted into open shadow.

The caveats are real:

• The init script must land before the framework caches a reference to Element.prototype.attachShadow. Some polyfills and bundlers grab the reference at module-load time and defeat the patch.
• App code that re-attaches a root via a stashed reference bypasses the patch.
• A CDP fallback (DOM.querySelector with pierce: true) exists for inspection but Playwright does not surface it in Locator; you would drop to context.newCDPSession for read-only access.

Real enterprise apps overwhelmingly avoid closed shadow. Salesforce LWC uses synthetic shadow (a polyfill, fully queryable) by default; native mode uses mode: 'open'. SAP UI5 Web Components and ServiceNow Now Experience both use open shadow. Closed-mode shadow is largely a worst-case demo construction; in production, the real friction with web-component-heavy frontends is deep shadow nesting and framework-specific selector conventions, not the shadow seal itself.

Verdict: solvable with one extra frameLocator call per chain. FrameLocator supports the full getBy* API and is a first-class chainable locator in Playwright.

The frame-aware version of this test:

const frame = page.frameLocator('iframe[title="login-frame"]');
await frame.getByLabel('Email').fill('user@example.com');
await frame.getByLabel('Password').fill('hunter2');
await frame.getByRole('button', { name: 'Sign in' }).click();
await expect(frame.getByText('Access granted')).toBeVisible();

The cost is one frameLocator() call at the top of each test that touches embedded content. That's modest enough to belong in the 2/5 band.

Real-world same-origin iframes in 2026 are rarer than the page implies. The classic payment / SSO examples (Stripe Elements, Adyen Web Drop-in, Braintree Hosted Fields, Auth0 Universal Login) are cross-origin by design for PCI isolation; that's a different problem covered under the cross-origin iframe row. Genuine same-origin iframe surfaces today are legacy WYSIWYG editors (TinyMCE / CKEditor classic), web-mail composers (Gmail, Outlook Web), and legacy intranet portals served from the same parent domain.

Verdict: impossible from inside Playwright. Playwright can drag (mouse.down/move/up at coordinates) but it cannot SEE the target zone.

The only paths a Playwright author has:

• Take a screenshot via Playwright, pass the image to an external OCR / template-matching service, extract the target X coordinate, dispatch page.mouse.down/move/up. At that point you have built half of AIVA inside your test runner.
• Use a paid CAPTCHA-solving service (similar to the Turnstile case). The service routes the challenge through real browsers, returns the solved token. Per-run cost + third-party dependency.

For real production slider CAPTCHAs (GeeTest, Alibaba, AWS WAF), the gap position is also rotated, scaled, and obfuscated with noise — generic OCR fails. Vendor-specific solver services are the only working option, and they cost ~$1-3 per 1000 solves.

Verdict: only brittle structural fallbacks remain. The promise of accessibility-driven testing is gone here.

Every label is an SVG/PNG image with empty alt. Playwright's accessibility-based locators return empty:

• page.getByLabel('Email') — no <label> element exists.
• page.getByRole('textbox', { name: 'Email' }) — no accessible name on the input.
• page.getByPlaceholder('Email') — no placeholder.
• page.getByText('Email') — text is in image pixels, not a DOM text node.

Possible fallbacks:

1. page.locator('input').nth(0) — works this layout; breaks on the slightest reorder.
2. Click at hard-coded pixel coordinates via page.mouse.click(x, y) — exactly the kind of brittle, screen-resolution-dependent code that motivates moving away from selector tests in the first place.
3. Integrate an OCR library, OCR the screenshot, find the label position, derive coordinates — at which point your test suite has reimplemented visual automation badly.

In real production deployments (bank login keypads with shuffled-position digit images), each session also changes the layout — so even nth-child fallbacks decay across runs.

Verdict: solvable in Playwright via frameLocator. Same-origin policy restricts scripts running on the parent page, not the automation driver itself.

Playwright talks to the browser via its own protocol (CDP+ in Chromium), and Chromium reassigns the tracked session to an out-of-process iframe when one is detected. So page.frameLocator(...), page.frame(...), and locator.contentFrame() all work across origins. Stripe ships official Playwright testing patterns for filling card fields inside their iframe:

await page
  .frameLocator('iframe[name^="__privateStripeFrame"]')
  .locator('[data-elements-stable-field-name="cardNumber"]')
  .fill('4242424242424242');

The friction is real but not categorical:

• Test authors must know which iframe holds the target field (selectors like iframe[name="*"] or positional indices).
• Stable inner selectors are vendor-supplied — Stripe gives data-elements-stable-field-name; not every widget does.
• The arena's data: URI variant is harder than typical cross-origin iframes because the opaque origin defeats URL-based frame matching, but selector-based matching still works.
• Network-response interception across out-of-process iframes has known limitations (see #20809) — relevant if the test needs to inspect the iframe's traffic.

The real-world examples need correction. Auth0 Universal Login is not embedded as an iframe in production — Auth0 sets X-Frame-Options: deny, so the login flow is a top-level navigation to *.auth0.com that Playwright fills directly. Cloudflare Turnstile is a cross-origin iframe, but the friction is fingerprinting + behavioural scoring + server-side token verification, not the iframe boundary — that belongs in Bot Detection level 5. Stripe Elements is the one genuine cross-origin iframe case from the original list, and Playwright handles it routinely.

Verdict: solvable with a per-list scroll-and-wait helper. Documented in vendor docs (AG Grid publishes an official Playwright E2E guide; LSEG maintains an open-source ag-grid-playwright bridge) and in Playwright issue threads.

The canonical pattern for each virtualised list a test interacts with:

1. Scroll the inner container with page.evaluate(el => el.scrollTo(0, y), container) or use locator.scrollIntoViewIfNeeded() on a row anchor.
2. waitFor the target row to mount, then act on it.
3. If the list exposes keyboard navigation (react-window, TanStack Virtual), keyboard.press('ArrowDown') often works as a portable alternative.

The friction (3/5, not 4/5) lives in the rough edges: locator.count() reports only mounted rows so size assertions need a workaround (#17042); virtualisation libraries (react-window, TanStack Virtual, AG Grid) expose different scroll APIs so the helper is per-library; some apps virtualise rows AND columns. But the recipes are well-documented and the AG Grid official guide ships a setupAgTestIds helper that turns this into a solved problem.