When AI changes software, the next question is how to verify the result. Perplexity's use of Astra draws attention to the work of building those tests, too.

A workflow can pass from start to finish while some external services are simulated. That creates a boundary around what was actually checked. What belongs in automated application tests, and what needs a separate evaluation?

The useful distinction is between responses prepared for a test and answers produced by a real model.

A test harness gives Astra a way to check application behavior

Perplexity cofounder Johnny Ho describes asking Astra to build a small testing program around an application. It simulates responses from services such as model APIs or connectors to exercise the workflow. He reports checking in less frequently than with earlier models; no reduction rate or comparison conditions are provided. OpenAI customer story

This surrounding program, a test harness, supplies inputs and observes results. The official GPT-6 Astra model documentation lists coding and complex reasoning among its uses. Those capabilities and the suitability of a particular test's acceptance criteria still require separate judgments.

For an answer display, a useful criterion might be whether the user can read the received answer, rather than whether an internal function ran. Playwright's testing guidance similarly favors observable behavior over implementation details. Stating the expected outcome first makes generated tests easier to review.

Mocking an API puts its real behavior outside the test

Existing tools can substitute prepared responses for external APIs. The official Playwright API mocking example uses page.route() to intercept a browser request and route.fulfill() to return JSON without contacting the actual API. This illustrates the mechanism; it does not establish that Perplexity uses Playwright.

Fixed responses let you inspect application behavior under the same input conditions. They cannot establish whether the real service supplies that response, accepts authentication, or generates a correct answer.

The following is a test-design example, not a description of Perplexity's architecture.

Test arrangement	Question it addresses	Question left open
Run the application with fixed external responses	Does the interface or workflow handle this response as expected?	Can the real service supply it?
Exercise the workflow against the real service	Can this environment connect and handle the result under these conditions?	Does answer quality hold for other inputs?
Compare real model output on evaluation data	How well does it meet task-specific criteria?	How does it behave outside the evaluated cases?

Identifying the substituted dependencies makes a passing result more informative than the label “end-to-end” alone.

Reduce oversight by preserving acceptance criteria and failure evidence

Generative AI can produce different outputs from the same input. OpenAI's evaluation design guide recommends defining objectives, datasets, and metrics, then evaluating continuously as the system changes. This explains why tests with fixed responses need a companion process that assesses actual answers against task requirements.

For adoption, a reasonable starting point is one workflow with explicit success and failure criteria. A service failure, for example, might require that the application never display a completed state. This is a proposed design choice, not a procedure reported in the customer story.

Preserve evidence that makes failures traceable. For browser tests, Playwright's CI debugging guidance describes using Trace Viewer to inspect the action timeline, DOM snapshots, and network requests.

When assessing the benefit, record investigation and correction time alongside the number of check-ins. That makes the work left after delegation visible. Expand AI-assisted testing from the scope whose acceptance criteria and simulated components you can explain. That gives you a practical way to reduce oversight while retaining a basis for judging the result.

Source

• Title: Perplexity trusts GPT-6 Astra with end-to-end systems
• URL: https://openai.com/index/perplexity-improving-accuracy-with-astra