AWS benchmarks OpenAI models by cost per correct outcome instead of token price alone
The reproducible harness measures accuracy, agent trajectories, and accepted deliverables while documenting configuration and sample-size limits.
Key takeaways
AWS published an open-source benchmarking approach on September 11 for comparing GPT-5.6 Luna, Terra, and Sol on Amazon Bedrock with GPT-5.4 mini and nano on the OpenAI API. Instead of ranking models by dollars per million tokens, the harness measures cost per correct answer, cost per passing multi-turn research result, and cost per rubric-accepted professional deliverable. It uses one Responses API client and records timestamped result files, frozen grading prompts, and chart inputs. The post also documents important limits: sample sizes range from 48 to 198 items, Bedrock models ran with reasoning disabled, and API baselines used their defaults. AWS describes this as a practical configuration comparison rather than an estimate of intrinsic capability. Teams should rerun the method on 50 to 100 of their own tasks and include failures, retries, turns, latency, review, and rework in the decision.
Direct answer
AWS published an open-source benchmarking approach on September 11 for comparing GPT-5.6 Luna, Terra, and Sol on Amazon Bedrock with GPT-5.4 mini and nano on the OpenAI API. Instead of ranking models by dollars per million tokens, the harness measures cost per correct answer, cost per passing multi-turn research result, and cost per rubric-accepted professional deliverable. It uses one Responses API client and records timestamped result files, frozen grading prompts, and chart inputs. The post also documents important limits: sample sizes range from 48 to 198 items, Bedrock models ran with reasoning disabled, and API baselines used their defaults. AWS describes this as a practical configuration comparison rather than an estimate of intrinsic capability. Teams should rerun the method on 50 to 100 of their own tasks and include failures, retries, turns, latency, review, and rework in the decision.
Verified facts
The open-source harness uses one Responses API client to compare five models across single-call accuracy, multi-turn web-research trajectories, and rubric-graded professional deliverables.
AWS reports samples of 48 to 198 items. Bedrock GPT-5.6 models ran with reasoning disabled, while OpenAI API baselines used defaults; the results compare configurations, not intrinsic capability.
The method divides spending on both successful and failed attempts by successes, while recording agent turns, cumulative input tokens, latency, grading-prompt hashes, and timestamped result files.
What changed
- The cost unit shifts from token price to an outcome that clears a quality gate
- Agent turns and repeatedly transmitted context enter the total cost
- Failed attempts, review, and rework are included in model selection
- Model, region, price, and evaluation settings are preserved for reproducibility
Impact for AI users
The same nominal token price can produce very different success rates and turn counts, so a cheaper model may not deliver the lowest business cost. The methodology also warns against treating small gaps in a vendor sample as definitive: task distribution, reasoning settings, regional latency, pricing date, and the grader can all change the ranking. Before procurement or migration, teams should define an accepted outcome and then calculate the full cost using real failures and human rework.
Operating checklist
- Select 50 to 100 real tasks with known-good answers or explicit rubrics, then freeze the dataset and pass threshold.
- For every candidate, record the model snapshot, provider, region, reasoning settings, price date, output cap, and tool versions.
- Measure success rate, cost per accepted outcome, agent turns, tail latency, human review, and rework time together.
- Preserve raw responses, grading evidence, and failure categories, then rerun the same protocol when prices, models, or workload mix change.
AI frontier news and analysis, AI software and model tools, AI skill tutorials and validation methods, and AI account and permission guidance
FAQ
Why is token price insufficient for model selection?
Error rate, output length, and agent turns multiply total spend. A production workload buys accepted outcomes rather than isolated tokens.
Can the AWS results choose a model for every organization?
No. The authors explicitly recommend rerunning the harness on your own tasks and disclose sample-size, configuration, and timing limits.
Is the cross-platform result an intrinsic capability ranking?
No. Bedrock models ran with reasoning disabled while API baselines used defaults, so the post compares the tested deployment configurations.
Summary
An actionable model-cost evaluation defines an accepted outcome first, then accounts for failures, turns, latency, and rework. The vendor benchmark supplies a method and starting point; evidence from the team's own workload must determine the final choice.
This AI-assisted article is checked by ENHE AI automation for official sources, bilingual fields, media rights, page safety, and historical duplication before publication.
What this means for everyday users
The same nominal token price can produce very different success rates and turn counts, so a cheaper model may not deliver the lowest business cost. The methodology also warns against treating small gaps in a vendor sample as definitive: task distribution, reasoning settings, regional latency, pricing date, and the grader can all change the ranking. Before procurement or migration, teams should define an accepted outcome and then calculate the full cost using real failures and human rework.
Tools you may use
Related tutorials
Related Tools And Tutorials
Use the following ENHE AI sections to continue from the news signal into tool selection, account-service guidance, or practical learning.
Related reading
OpenAI launches the Agents API public beta with the Codex harness and hosted sandboxes
OpenAI introduced the Agents API in public beta on September 10, exposing the harness and infrastructure behind Codex to developers. A single API call can define an agent task, model, tools, and environment. Teams may run it in an OpenAI-hosted sandbox, on their own infrastructure, or with listed ecosystem-provider integrations. OpenAI says the API supports MCP, custom functions, built-in tools such as web search, automatic context compaction for long sessions, and optional parallel subagents. The hosted sandbox can be configured with files, packages, skills, and plugins. The API is available to all developers in public beta with no separate API fee beyond token and tool charges. Teams should validate permissions, tool behavior, observability, and recovery paths before treating beta workflows as production-ready.
A voice-agent release checklist for VAD, noise, interruptions, transcription, tools, and failure handoffs
A voice-agent release should not be accepted after one successful conversation in a quiet room. Test cases need to cover speech-activity detection start and stop behavior, silence and background noise, user interruption, transcription evidence and correction, spoken status around tool calls, and state recovery after timeouts or failures. The OpenAI Realtime API reference documents capabilities for real-time sessions, audio input and output, transcription, and server-side voice activity detection. This article turns those capabilities into a general acceptance checklist. It is not a release announcement, a claim about a particular product version, or a performance guarantee. Teams should define their own language, device, privacy, latency, escalation, and human-review criteria before deployment.
AWS connects AgentCore evaluations to GitHub Actions for pre-merge agent regression gates
AWS published a reference workflow on September 8 for integrating Amazon Bedrock AgentCore Evaluations with GitHub Actions. The pipeline deploys a development agent and an OAuth-protected MCP server, invokes representative prompts, collects OpenTelemetry traces, and scores behavior before allowing a pull request to proceed. AWS describes built-in dimensions such as helpfulness, correctness, goal success, tool selection, tool parameters, and trajectory order, with custom and code-based evaluators available. The post compares three authentication patterns: evaluating stored traces, using a pre-authorized test user, or issuing machine-to-machine credentials for CI. Its implementation uses the third pattern. Teams should validate that CI credentials cannot escape their intended environment, that evaluation samples cover important failures, and that score variance, latency, evaluator cost, and rollback behavior are visible before treating the threshold as a release gate.
OpenAI reaches its automated research intern milestone while keeping human decision gates
OpenAI published an internal view of research acceleration on September 6, saying it has reached the automated research intern milestone announced last year. Researchers are using coding agents more often and in concurrent sessions, contributing code faster and running more experiments. OpenAI says August 2026 was the highest month for experiments per active experimenter since tracking began in January 2025, while noting that compute growth also affects the result. The company keeps people responsible for research priorities, interpreting results, and decisions to scale, pause, or deploy. The practical lesson is to measure automation at each step without confusing local throughput gains with total research progress or safe autonomous science.
OpenAI launches Daybreak for Frontline Defenders with a planned billion commitment
OpenAI announced Daybreak for Frontline Defenders on September 3, with a planned billion commitment for access subsidies, training, technical support, and partner programs. The initiative prioritizes water and wastewater utilities, power operators, state and local governments, community banks, nonprofits, and open-source maintainers. Supported work includes legacy-code review, suspicious-activity analysis, vulnerability discovery, and tested remediation. OpenAI also described a public-sector and water-system pilot with MS-ISAC and a Defense Network of more than 35 products and partners. The announcement suggests that frontier AI defense value depends on an operating network of authorization, monitoring, and support rather than model access alone. This gives teams a practical comparison point for deployment planning.
OpenAI Extends Zero Data Retention to Frontier Models with Private Safety Processing
OpenAI announced Private Safety Processing on August 19, 2026 to keep frontier-model safety monitoring compatible with Zero Data Retention for eligible API customers. Under the preview, customer content can remain on customer-controlled infrastructure, while automated systems analyze related interactions and return narrowly defined risk signals without giving OpenAI personnel the underlying prompts or responses. OpenAI is also developing hosted storage encrypted with customer-controlled keys. This is not an automatic setting for every API account. Teams should confirm organization and project eligibility, separate request content from administrative audit logs and other data surfaces, document legal or safety exceptions, and test the configuration with non-sensitive samples before sending regulated or confidential workloads.
