From Scripts to Goals: What Agentic QA Actually Changes for SDETs

The ticket landed in the backlog with a simple description: “Make sure the checkout flow works.” No reproduction steps, no specific assertions, no defined pass criteria. Just a goal.

A traditional automation framework cannot run that ticket. It runs scripts. It clicks the button you told it to click, asserts the text you told it to assert, and marks the test passed or failed based on criteria you defined in advance. Give it a goal and it has nothing to execute.

An agentic QA system can run that ticket. It reads the goal, decides which paths through the checkout flow to exercise based on risk and recent changes, generates the steps to execute each path, runs them against the live application, evaluates the results against what “works” means in context, and flags the paths that did not behave as expected. It made decisions you used to make, at a speed you cannot match when you are also doing twelve other things.

This shift from scripts to goals is what most of the “AI will replace QA” conversation is actually about, and it deserves a grounded look. What does Level 1, 2, and 3 automation actually mean? What changes for SDETs when the industry moves up the levels? What stays the same? And what can you do this quarter to position yourself for the shift rather than be caught by it?

Level 1: Scripted Automation

Most teams doing “automation” today are at Level 1. An SDET writes a script that executes a fixed sequence of actions: navigate to this URL, fill in these fields, click this button, assert that this text appears. The script is deterministic, reproducible, and completely brittle to changes in the sequence or the UI it depends on.

Level 1 automation is valuable. It catches regressions in the specific paths it covers, it runs faster than a human can, and it runs at 3 AM when nobody is awake. The 350-test API suite mentioned in an earlier post on this blog is Level 1. The Playwright UI tests that cover the happy path through checkout are Level 1. There is nothing wrong with Level 1 — it is the foundation that everything else builds on.

The limitation is that Level 1 covers what you scripted and nothing else. A requirement that was not translated into a script is not tested. A path through the application that a developer introduced during a refactor and nobody thought to add to the suite is not tested. A combination of conditions that only occurs in production because of real user behavior is not tested. The coverage is bounded by the imagination and bandwidth of the person writing scripts.

The other limitation is maintenance. Level 1 scripts encode the implementation details of the application at the time they were written. When the implementation changes, the scripts break. Selector maintenance, response format changes, flow restructuring, these are all Level 1 taxes that grow as the suite grows and compound as the application evolves.

Level 2: Orchestrated Agents

Level 2 is where most forward-looking teams are trying to get to in 2026. An orchestrated agent is a system that combines a language model’s reasoning with a set of tools it can call, operating under instructions from a human to accomplish a defined task.

The difference from Level 1 is that the agent decides the sequence of actions rather than executing a predefined sequence. A Level 2 agent given “test the password reset flow” will read the feature spec if one is available, reason about the paths through the flow, decide which ones to exercise first based on risk, generate the steps for each path, execute them, evaluate the outcomes, and report what it found. It may discover a path you did not script, because it is not constrained to the scripts that already exist.

At Motive, the RCA agent described elsewhere on this blog is a Level 2 system. It receives a task (“analyze this failing incident”), reasons about what data it needs, calls MCP tools to retrieve that data from Jira and Glean and GitHub, synthesizes a root-cause analysis, and produces a structured artifact for human review. The human hands it a goal. The agent figures out the steps.

The SDET’s role at Level 2 shifts significantly. You spend less time writing individual test scripts and more time designing the tools the agent can call, writing the instructions that define what “done” looks like for a task, and reviewing the agent’s output to verify that its reasoning was sound. You are building and evaluating an automated QA process rather than executing one.

This shift requires a different skill set. Understanding what makes a good tool definition, how to write agent instructions that produce consistent output, and how to evaluate whether an agent’s test results are trustworthy are all skills that matter at Level 2 and barely matter at Level 1.

Level 3: Goal-Directed Agents

Level 3 is the frontier. A goal-directed agent receives an objective, like “ensure the checkout flow is working correctly before this release,” and determines autonomously what testing is needed, designs the test strategy, executes it, interprets the results, and decides whether the release is safe to proceed.

There are glimpses of Level 3 behavior in the tools available today. Computer-use agents that can navigate real applications without predefined selectors. Agents that propose their own test cases based on a requirements document and execute them without human scaffolding. Multi-agent systems where a planner agent delegates subtasks to specialist agents and synthesizes their outputs into a coherent quality report.

Level 3 is not here yet in the form that would let you hand off a release decision entirely. The gap is mostly about trust calibration and failure recovery. A Level 3 agent needs to know when it is uncertain, ask for guidance rather than guess, and recognize that a scenario it did not anticipate may require human judgment before acting. Current systems are inconsistent on all three dimensions. They are capable in familiar territory and unreliable in novel situations, and the novel situations are exactly the ones where the judgment call matters most.

But the direction is clear, and Level 3 is worth understanding now because the engineers who understand goal-directed systems, who know how to define objectives precisely enough for an agent to act on them, who know how to evaluate whether an agent’s conclusion is trustworthy, will be the ones who shape how Level 3 gets deployed when it arrives.

What Changes for SDETs

The shift across levels changes what work SDETs do, in some areas reducing the low-level scripting work and in others creating new demands that did not exist before.

What decreases: time spent writing individual test cases for well-defined features, selector maintenance for stable UI flows, debugging deterministic test failures that have clear causes, copy-paste work of extending an existing suite to cover a new variation of a known scenario. These are exactly the kinds of tasks that Level 2 agents handle well, because they are structured, repetitive, and bounded.

What increases: designing the testing strategy that the agent executes, writing the tool definitions that give the agent access to the right data, building the evaluation criteria that tell you whether the agent’s output was trustworthy, and reviewing the agent’s conclusions on the tasks that carry real risk. These require deeper domain knowledge, not less, because you are deciding what matters rather than scripting what to check.

What is new entirely: working with non-deterministic systems. At Level 1, a test either passes or fails and the reason is usually findable. At Level 2 and 3, the agent’s behavior is probabilistic, its decisions can be wrong in subtle ways, and evaluating whether its output is correct requires the same judgment that evaluating a junior engineer’s work requires. SDETs who develop this judgment, who can read an agent’s reasoning and recognize when it is sound versus when it is confidently wrong, will be significantly more effective than those who treat agent output as authoritative.

What Stays the Same

The things that do not change are the things that have always mattered most in QA.

Domain knowledge. An agent that does not understand what “correct” means for a checkout flow in your specific business context will miss the things that matter and flag things that do not. You carry that context. The agent borrows it through the instructions and tools you provide.

Adversarial thinking. The instinct to ask “how could this go wrong?” is not something current agents do well in novel situations. They test the scenarios they are instructed to test and the scenarios that follow predictably from their training. The edge cases that only a human who has seen the system fail in production knows to look for still require human instinct.

Risk prioritization. Deciding which part of the application needs the most rigorous testing before a release, given what changed and what the business impact of a failure would be, requires judgment that combines technical and business context. Agents can surface data to inform that decision, but the decision itself still belongs to a person who is accountable for it.

Communication. Translating a quality signal into something a product manager or an engineering lead can act on is a skill that only gets more important as the automation layer gets more complex. “The agent flagged eleven things, here is the two that matter and here is what to do about them” is valuable in proportion to how much the agent flagged.

Moving From Level 1 to Level 2 This Quarter

The practical path from Level 1 to Level 2 does not require rebuilding your test suite. It requires adding an orchestration layer on top of what you already have.

Start by identifying one high-frequency, well-defined QA task that a Level 2 agent could handle: CI failure triage, regression test selection for a given diff, first-pass root cause analysis for a flaky test. Pick a task where the current human process is mostly information retrieval and synthesis rather than novel judgment.

Build or configure the MCP tools the agent needs for that task. If the task is CI triage, the agent needs to read build logs, list failed tests, and check test history. If the task is flaky test diagnosis, it needs to query recent test runs and check whether a test failed consistently or only occasionally under specific conditions. The tools are the hard part; the agent instructions are comparatively easy once the tools exist.

Run the agent on ten or twenty real instances of the task before deploying it as the default workflow. Compare its output to what a human engineer would have concluded. Note where it is wrong or incomplete. Use those notes to refine the tools, the instructions, or both. Treat the iteration as product development rather than prompt tuning.

The teams that are already at Level 2 on significant parts of their quality process did not get there by replacing their existing automation. They got there by adding an orchestration layer above it, with the scripted automation as the execution substrate and the agent as the planning and evaluation layer. That architecture is available to any team with a functioning test suite and forty hours of focused engineering work.

The Role Is Not Going Away

The fear underneath the “AI will replace QA” conversation is understandable. Automation has historically displaced the most scripted, most repetitive parts of many jobs, and QA has a lot of scripted, repetitive work.

What the fear gets wrong is the difference between tasks and judgment. Agents are getting very good at the tasks: running test cases, triaging failures, generating coverage for well-defined scenarios. They are not getting good at the judgment: deciding what to test, deciding whether a result is trustworthy, deciding when a risk is acceptable and when it is not. The judgment is what makes a QA engineer valuable, and it is also what makes good QA engineering hard enough that teams keep under-investing in it.

The SDET who automates their own scripting work, who builds agents that handle the repetitive 80% of their task load, and who redirects their time toward domain knowledge, adversarial thinking, and risk judgment is not at risk of being replaced. They are the person building the system that replaces the scripting work, and they are harder to replace than ever because they understand both the quality domain and the agent systems that operate within it.

The checkout ticket that started this post — “make sure checkout works” — is a goal. Goals are what SDETs have always been working toward. The scripts were just one way to get there. Agentic systems are another, faster way to cover more of the ground. Your job is to point them in the right direction and verify that they got there.