A Reproducible AI Game Creation Benchmark

This is a proposed test, not a leaderboard. No measurements have been collected for the scorecard below. Use it to compare tools on the same game and publish the evidence alongside any results.

The fixed task

Make a one-screen 2D dodge game. The player moves with keyboard controls, avoids moving hazards, earns one point per second and can restart after a collision. The game must show the score and a clear game-over state. Use the same written brief, target device, time limit and allowed assets for every tool. Record exact wording, tool and model versions, plan, date, hardware, browser and cost limits.

Run each trial

1. Start a fresh project and timer. Submit the fixed brief. Keep a log of prompts, errors, retries and manual edits; do not hide failed attempts.
2. At 30 minutes save the project and record whether each required behavior works. At 45 minutes request the fixed revision below, then stop editing at 60 minutes. Record the change request and test again, including restart and scoring. Note elapsed human time separately from waiting time.
3. Have a second person try the delivered build on the declared target. Record build steps, access failures and whether they could start and restart without help.
4. Record total observed charges or credits, plan restrictions and available project files. Mark unavailable values as unknown; distinguish a quote from a paid invoice.

Blank scorecard

Count each of movement, hazards, scoring, loss and restart once only if a documented test passes. Report each dimension separately rather than combining cost and quality into an arbitrary score. Repeat trials in alternating order; disclose the operator’s familiarity and all manual work. A single trial is illustrative, not a general ranking.

Keep evidence usable

Save the prompt log, timestamped builds, screenshots or recordings of each test, costs and a defect list. Record missing evidence and permission to share assets. A hosted link can change; preserve a version or record that it could not be archived. Before testing, consult the product’s own terms: Rosebud pricing, Summer MCP setup and Godot export documentation describe different constraints.

Exact brief and controlled inputs

Copy this brief unchanged for each trial: “Make a single-screen 2D dodge game for a desktop browser. The player is a visible shape controlled with arrow keys, stays inside the play area, and avoids moving hazards. Start at zero and add one point for each full second survived. On contact with a hazard, stop the game and show the final score and a game-over message. Provide a restart control that resets score and returns to a playable game. Use simple shapes and no external assets.” Do not add solutions, code or screenshots unless they are available to every candidate and recorded as a separate condition.

Before starting, set the same screen size, browser, device, network and account tier where possible. Record any mismatch. Use a new empty project for each run, the same operator, and the same 60-minute wall-clock limit. The operator may use documentation and manual edits; record every action, including copied code, because the test measures a workflow rather than an isolated model. Do not reuse a previous trial’s generated files or fixes.

Timing and iteration rules

Start the clock at the first submitted brief. Allow at most ten additional requests, including retries and the revision request. Number each request and save the full text and response. Manual actions are allowed inside the same 60 minutes and do not consume a request; log the action and its start and end. A failed tool request still counts. If the product imposes a smaller limit, stop and record that constraint instead of buying extra capacity for only one candidate.

At 30 minutes, freeze a copy or record the immutable version, run the five initial checks below and log the result. Continue until the five checks pass or the clock reaches 45 minutes. At that point submit exactly: “After the score reaches 20, hazards move twice as fast. Keep the other rules unchanged.” Spend the remaining time on that change and repairs within the request limit. At 60 minutes stop editing and preserve the final version. If a checkpoint interrupts generation, record the state and time; do not silently extend the limit.

Five observable checks

1. Movement: hold each arrow key for two seconds. The player moves in the expected direction and remains inside the play area.
2. Hazards: observe at least one visible hazard moving on screen. The player can encounter it through normal play.
3. Score: survive ten seconds measured with a separate timer. The displayed score increases by ten, allowing one point of timing tolerance.
4. Loss: touch a hazard. Play stops, the game-over message appears and a final score is visible.
5. Restart: activate restart after loss. Score returns to zero, controls work and a new collision produces another loss.

Pass a check only if all its stated conditions are observed in the same saved version. A crash, missing control, unexplained blocking dialog or untestable behavior is a failure; write the reason. Mark a check “not run” only when evidence is unavailable, and do not count it as a pass. For the revision, record hazard speed before and after 20 points using the same visible distance and timer or a recording. If that cannot be measured reliably, mark the revision inconclusive. Repeat all five checks for regressions.

Evidence sheet for each run

Keep a recording showing the timer, game and checks where possible. Save prompts and responses in order, a copy or version reference at each checkpoint, and a defect log with steps to reproduce. Do not publish private project data or assets without permission. If the service provides only a mutable link, say so; another reader cannot assume it still represents the tested version.

Interpretation and limitations

Do not award a full pass for a game that looks right but cannot be played. Do not count a paid plan price as actual marginal cost or extrapolate a single run into a monthly bill. A tool can complete the task and still fail the independent delivery check. Show the five checks, revision, delivery and cost separately so a reader can apply their own priorities.

How many runs are enough?

One run is a worked example. For a comparison, perform at least three fresh trials per tool, alternating the order to reduce learning effects, and publish all runs rather than the best one. This is still a narrow experiment with one operator and one task, not proof of general product performance.

What if a tool cannot target a desktop browser?

Record the target mismatch and do not claim an equivalent result. You can run a separate native-target experiment with the same gameplay checks, but label its target and conditions and avoid combining it with the browser results.

Explore Summer Engine ↗

Looking for something to play? Discover games on Summer Games.