AWS Let Agents Build a Cloud Emulator With No Human Code Review. The Review Moved Into the Tests

November 13, 2026
8 min read

November 13, 2026
8 min read
On 11 November AWS released AWS Emulator, a local stand-in for its services that you can run on your laptop to build and test applications without calling the cloud. The tool is useful, but the claim that came with it matters more to founders. In the words of Matthew Goldberg, the senior principal engineer who wrote the announcement: "The emulator was written entirely by autonomous agents. Our team hasn't written or reviewed any of the code." Most startups have spent this year finding that human review is the slowest step in an agent-heavy team. AWS has shown one way to remove it, and the conditions that made that possible are the useful part.
AWS describes the product simply: "a local emulator of AWS services that you can run on your machine to build and test your AWS applications." Coverage adds the details:
AWS says the emulator "is fundamentally built by three teams of agents. One team manages a test suite; one team finds issues in the emulator; and one team fixes those issues." A fourth stage decides what ships.
The scale is large. AWS says "hundreds of agents concurrently work on the codebase" and that it has "seen thousands of issues fixed every day". The project began as a weekend experiment: an empty web server that accepted requests for every API model "but did nothing", with the test suite as the specification.
Nobody removed the review. AWS moved it from a person reading diffs to a test suite checked against the real system.
AWS's post gives no pass rates; it promises a later post on how fidelity is measured. Help Net Security reports that AWS publishes per-service fidelity scores with each release, and i-SCOOP counted them for the launch release. Its results:
Read those numbers with i-SCOOP's own caveat: "a test only passes if the emulator returns exactly what AWS returned", and "a failing test often means one wrong error message, not a broken service." Its summary is fair: a fast, free emulator "that gets the common paths of the popular services mostly right."
The limitations are clearer than the scores. Help Net Security reports that the emulator "does not evaluate AWS Identity and Access Management (IAM) or resource-based policies as AWS does", keeps everything in memory, and should be used with one region and one account. i-SCOOP adds that a request AWS would deny can succeed locally, and the reverse.
One condition did most of the work: AWS had an oracle. For any request, real AWS gives the correct answer, and that answer can be recorded once and replayed thousands of times for free. An agent's change is not judged by whether a reviewer likes the diff. It is judged by whether the emulator now matches the system it copies. That is a stronger check than most human review, and it never gets tired.
Most startup code has no such oracle. Nobody can record the correct answer for "is this onboarding flow good" or "is this pricing page right". But many parts of a product do have one, and those parts can move toward the AWS model:
import json, pathlib
CASES = pathlib.Path("fidelity_cases")
def record(name: str, request: dict, call_real) -> None:
"""Run once against the real system and save its answer."""
response = call_real(request)
CASES.joinpath(f"{name}.json").write_text(
json.dumps({"request": request, "expected": response}, indent=2))
def replay(call_ours) -> dict:
"""Run every saved case against our code; no calls to the real system."""
results = {"pass": 0, "fail": []}
for path in sorted(CASES.glob("*.json")):
case = json.loads(path.read_text())
if call_ours(case["request"]) == case["expected"]:
results["pass"] += 1
else:
results["fail"].append(path.stem)
return results
def gate(before: dict, after: dict) -> bool:
"""Merge only if nothing that passed before fails now."""
newly_failing = set(after["fail"]) - set(before["fail"])
return not newly_failing
Find your oracles. List the parts of your codebase where a correct answer exists outside your code: a vendor API, the old system, a reference dataset. Those are the parts where agents can work with the least human review.
Record before you delegate. Capture the real system's answers first, as AWS's test writers did, and have each one checked against the source. A test an agent writes without checking it against the real thing only confirms what the agent already believes.
Gate on regressions, not on taste. Merge only when the build passes, standards and security checks pass, and nothing that passed before now fails. That rule is cheap to run and hard to argue with.
Publish your own score. Track the pass rate per area and show it to the team. A number that admits 56.5% is more useful than a claim of completeness, because people can see where to trust the code.
Keep humans where there is no oracle: product judgment, security boundaries and anything involving permissions. AWS's own result points the same way. The one area the emulator does not try to match, access control, is the area where a wrong answer is most dangerous. In July we argued that a startup should hire the reviewer before the coder. AWS Emulator refines that advice: hire the reviewer for the parts without an oracle, and spend the rest of the effort building one. For the parts of an AI product where you can define correct behaviour, start with an evaluation set of real user tasks.
If you plan to use AWS Emulator
Treat it as a fast local loop, not proof that your code works on AWS. Do not use it to test access control, because IAM and resource policies are not evaluated the way AWS evaluates them. Use one region and one account, expect state to disappear when it stops, and check the per-service score for the services you depend on. i-SCOOP also warns that if your SDK is not pointed at the emulator, it "quietly calls real AWS with whatever credentials it finds." Run your final checks on real AWS.
That no human was involved. No human wrote or reviewed the code, according to AWS. People still chose the oracle, designed the three teams, set the coding standards and built the merge gate. AWS credits "the guardrails we've put in place". The human work moved earlier in the process; it did not disappear.
That 56.5% means half the emulator is broken. The tests demand exact matches, including error messages. The common paths of popular services score far higher than the long tail.
That your product can do the same. AWS had a perfect, cheaply queried reference system and the budget to run hundreds of agents. Most startups have neither for most of their code. Copy the method where you have an oracle, not everywhere.
That the numbers are settled. AWS's announcement says both "thousands" and "hundreds" of agents, and the pass rates come from an outside count of one release, not from AWS. Expect both to change as AWS publishes more.
On 11 November 2026 AWS released AWS Emulator, a free local emulator of its services, and said its team had not written or reviewed any of the code. Agents wrote tests checked against real AWS, other agents replayed them to find divergence, a third group fixed each issue, and a merge gate checked build, standards, security and fidelity. An independent count of the launch scores found 56.5% of 42,937 fidelity tests passing, with wide variation between services and no evaluation of IAM policies.
For founders, the lesson is not that review is obsolete. It is that review can move into tests when you have a reliable source of correct answers, and should stay with people when you do not.
This week, pick one part of your codebase that has an oracle, record twenty real answers from it, and make "nothing that passed now fails" a merge rule for agent changes there.
Sources: Matthew Goldberg, AWS, AWS Emulator: a local emulator built 100% by autonomous AI agents (11 November 2026) — the product description, the 438 API models, the CC0 licence, the three teams of agents, the merge gate, the agent and issue counts, and the statement that the team had not written or reviewed the code; Anamarija Pogorelec, Help Net Security, AWS Emulator: Open-source tool for testing AWS apps locally, built by autonomous AI agents (12 November 2026) — the four ways to run it, the per-service scores published with each release, and the IAM, region, account and in-memory limitations; Tom Cuylaerts, i-SCOOP, AWS Emulator was built by AI agents nobody reviewed, and its own test scores show where to trust it (12 November 2026) — the count of the launch scores, the per-service figures, the caveats on exact matching, the note on pull requests and Apache-licensed vendored code, and the warning about SDK endpoints. The code in section 4 is my illustration, and the reading in sections 4 and 5 is mine.
IdeaToMVP Academy
4-week live cohort for founders. Learn to ship AI agents, scope MVPs, and automate your business — taught by the same team that writes these guides.