AWS Let Agents Build a Cloud Emulator With No Human Code Review. The Review Moved Into the Tests

Surya Pratap
By Surya Pratap

November 13, 2026

8 min read

AI & Technology
A two-part diagram titled 'Nobody removed the review. It moved into the tests.' On the left, under 'How agents built AWS Emulator', four steps: test writers record each test against real AWS; issue finders replay the tests against the emulator; fixers take one issue each and change the code; and a merge gate checks build, standards, security and fidelity. On the right, under 'What the published scores say', an independent count of the launch scores: 56.5% of 42,937 fidelity tests passing, 40 services at 90% or more, 124 services below 50%, and IAM policies not evaluated as AWS does. A footer gives the sources as AWS's announcement of 11 November 2026 and an analysis of its scores.See the process, and the scoresHover to explore
Agents could skip the human reviewer because every change was checked against the real thing.

On 11 November AWS released AWS Emulator, a local stand-in for its services that you can run on your laptop to build and test applications without calling the cloud. The tool is useful, but the claim that came with it matters more to founders. In the words of Matthew Goldberg, the senior principal engineer who wrote the announcement: "The emulator was written entirely by autonomous agents. Our team hasn't written or reviewed any of the code." Most startups have spent this year finding that human review is the slowest step in an agent-heavy team. AWS has shown one way to remove it, and the conditions that made that possible are the useful part.

This article is based on AWS's announcement of 11 November 2026, on Help Net Security's report of 12 November, and on an analysis by i-SCOOP that counted the published fidelity scores. AWS's own post gives no pass rates, so every score below is i-SCOOP's count, not an AWS figure. The reading from section 4 onward is mine.

1. What AWS shipped

AWS describes the product simply: "a local emulator of AWS services that you can run on your machine to build and test your AWS applications." Coverage adds the details:

  • Scope. AWS says the project was seeded with "438 AWS API Models". i-SCOOP puts that at more than 200 services, with scores published for 355.
  • How you run it. Help Net Security lists four ways in: through an AI coding agent, as a Linux package, in Docker, or embedded inside a Rust test process.
  • Licence. AWS released the code under Creative Commons CC0, a public-domain dedication. i-SCOOP notes that vendored code is under Apache 2.0.
  • Contributions. According to i-SCOOP, "The repository does not accept pull requests, only bug and fidelity reports." Outside contributions would break the claim that agents wrote every line.

2. How agents built it without a human reviewer

AWS says the emulator "is fundamentally built by three teams of agents. One team manages a test suite; one team finds issues in the emulator; and one team fixes those issues." A fourth stage decides what ships.

  1. Test writers record the truth. Agents write tests from the public AWS API models and "verify each one against real AWS before submitting." The results are saved as fidelity tests, now "tens of thousands" of them, that "can be rerun repeatedly without calling AWS again."
  2. Issue finders replay the tests. A second team runs the fidelity tests against the emulator, reasons about the failures and groups those that may share a cause.
  3. Fixers take one issue each. "Each issue is given to an agent in the third team, whose job is to fix that single issue." The agent checks out the code, finds the root cause and edits until the issue is fixed.
  4. A merge gate decides. The gate "obviously blocks changes that don't compile. But it also includes agents that review against coding standards, verify the code is secure, and ensure that the change did not regress fidelity." Changes that fail go back for rework.

The scale is large. AWS says "hundreds of agents concurrently work on the codebase" and that it has "seen thousands of issues fixed every day". The project began as a weekend experiment: an empty web server that accepted requests for every API model "but did nothing", with the test suite as the specification.

Nobody removed the review. AWS moved it from a person reading diffs to a test suite checked against the real system.

3. The scores are the honest part

AWS's post gives no pass rates; it promises a later post on how fidelity is measured. Help Net Security reports that AWS publishes per-service fidelity scores with each release, and i-SCOOP counted them for the launch release. Its results:

  • Overall. 24,243 of 42,937 fidelity tests pass, or 56.5%. At the level of individual steps, 73.2% pass.
  • Spread. Among 342 services with at least ten tests, the median pass rate is 61.3%. 40 services score 90% or more; 124 score below 50%.
  • Familiar services. EventBridge 71.5%, DynamoDB 68.1%, SQS 67.8%, S3 62.7%, Lambda 51.9%, EC2 38.8%, and CloudFormation 18.9% across 4,206 tests.

Read those numbers with i-SCOOP's own caveat: "a test only passes if the emulator returns exactly what AWS returned", and "a failing test often means one wrong error message, not a broken service." Its summary is fair: a fast, free emulator "that gets the common paths of the popular services mostly right."

The limitations are clearer than the scores. Help Net Security reports that the emulator "does not evaluate AWS Identity and Access Management (IAM) or resource-based policies as AWS does", keeps everything in memory, and should be used with one region and one account. i-SCOOP adds that a request AWS would deny can succeed locally, and the reverse.

4. Why it worked, and whether it works for you

One condition did most of the work: AWS had an oracle. For any request, real AWS gives the correct answer, and that answer can be recorded once and replayed thousands of times for free. An agent's change is not judged by whether a reviewer likes the diff. It is judged by whether the emulator now matches the system it copies. That is a stronger check than most human review, and it never gets tired.

Most startup code has no such oracle. Nobody can record the correct answer for "is this onboarding flow good" or "is this pricing page right". But many parts of a product do have one, and those parts can move toward the AWS model:

  • an integration that must match a third-party API's behaviour;
  • a migration or data pipeline whose output can be compared with the old system's;
  • a parser, importer or calculation with known correct results;
  • a rewrite of a service whose existing behaviour is the specification.
pythonRecord once against the real system, replay against yours
import json, pathlib

CASES = pathlib.Path("fidelity_cases")

def record(name: str, request: dict, call_real) -> None:
    """Run once against the real system and save its answer."""
    response = call_real(request)
    CASES.joinpath(f"{name}.json").write_text(
        json.dumps({"request": request, "expected": response}, indent=2))

def replay(call_ours) -> dict:
    """Run every saved case against our code; no calls to the real system."""
    results = {"pass": 0, "fail": []}
    for path in sorted(CASES.glob("*.json")):
        case = json.loads(path.read_text())
        if call_ours(case["request"]) == case["expected"]:
            results["pass"] += 1
        else:
            results["fail"].append(path.stem)
    return results

def gate(before: dict, after: dict) -> bool:
    """Merge only if nothing that passed before fails now."""
    newly_failing = set(after["fail"]) - set(before["fail"])
    return not newly_failing

Find your oracles. List the parts of your codebase where a correct answer exists outside your code: a vendor API, the old system, a reference dataset. Those are the parts where agents can work with the least human review.

Record before you delegate. Capture the real system's answers first, as AWS's test writers did, and have each one checked against the source. A test an agent writes without checking it against the real thing only confirms what the agent already believes.

Gate on regressions, not on taste. Merge only when the build passes, standards and security checks pass, and nothing that passed before now fails. That rule is cheap to run and hard to argue with.

Publish your own score. Track the pass rate per area and show it to the team. A number that admits 56.5% is more useful than a claim of completeness, because people can see where to trust the code.

Keep humans where there is no oracle: product judgment, security boundaries and anything involving permissions. AWS's own result points the same way. The one area the emulator does not try to match, access control, is the area where a wrong answer is most dangerous. In July we argued that a startup should hire the reviewer before the coder. AWS Emulator refines that advice: hire the reviewer for the parts without an oracle, and spend the rest of the effort building one. For the parts of an AI product where you can define correct behaviour, start with an evaluation set of real user tasks.

If you plan to use AWS Emulator

Treat it as a fast local loop, not proof that your code works on AWS. Do not use it to test access control, because IAM and resource policies are not evaluated the way AWS evaluates them. Use one region and one account, expect state to disappear when it stops, and check the per-service score for the services you depend on. i-SCOOP also warns that if your SDK is not pointed at the emulator, it "quietly calls real AWS with whatever credentials it finds." Run your final checks on real AWS.

5. What I would not claim

That no human was involved. No human wrote or reviewed the code, according to AWS. People still chose the oracle, designed the three teams, set the coding standards and built the merge gate. AWS credits "the guardrails we've put in place". The human work moved earlier in the process; it did not disappear.

That 56.5% means half the emulator is broken. The tests demand exact matches, including error messages. The common paths of popular services score far higher than the long tail.

That your product can do the same. AWS had a perfect, cheaply queried reference system and the budget to run hundreds of agents. Most startups have neither for most of their code. Copy the method where you have an oracle, not everywhere.

That the numbers are settled. AWS's announcement says both "thousands" and "hundreds" of agents, and the pass rates come from an outside count of one release, not from AWS. Expect both to change as AWS publishes more.

The honest summary

On 11 November 2026 AWS released AWS Emulator, a free local emulator of its services, and said its team had not written or reviewed any of the code. Agents wrote tests checked against real AWS, other agents replayed them to find divergence, a third group fixed each issue, and a merge gate checked build, standards, security and fidelity. An independent count of the launch scores found 56.5% of 42,937 fidelity tests passing, with wide variation between services and no evaluation of IAM policies.

For founders, the lesson is not that review is obsolete. It is that review can move into tests when you have a reliable source of correct answers, and should stay with people when you do not.

This week, pick one part of your codebase that has an oracle, record twenty real answers from it, and make "nothing that passed now fails" a merge rule for agent changes there.

Sources: Matthew Goldberg, AWS, AWS Emulator: a local emulator built 100% by autonomous AI agents (11 November 2026) — the product description, the 438 API models, the CC0 licence, the three teams of agents, the merge gate, the agent and issue counts, and the statement that the team had not written or reviewed the code; Anamarija Pogorelec, Help Net Security, AWS Emulator: Open-source tool for testing AWS apps locally, built by autonomous AI agents (12 November 2026) — the four ways to run it, the per-service scores published with each release, and the IAM, region, account and in-memory limitations; Tom Cuylaerts, i-SCOOP, AWS Emulator was built by AI agents nobody reviewed, and its own test scores show where to trust it (12 November 2026) — the count of the launch scores, the per-service figures, the caveats on exact matching, the note on pull requests and Apache-licensed vendored code, and the warning about SDK endpoints. The code in section 4 is my illustration, and the reading in sections 4 and 5 is mine.

IdeaToMVP Academy

Want to build with AI — not just read about it?

4-week live cohort for founders. Learn to ship AI agents, scope MVPs, and automate your business — taught by the same team that writes these guides.

Explore the Academy →
Share this post :