AWS benchmarks OpenAI models by cost per correct outcome instead of token price alone
The reproducible harness measures accuracy, agent trajectories, and accepted deliverables while documenting configuration and sample-size limits.
Key takeaways
AWS published an open-source benchmarking approach on September 11 for comparing GPT-5.6 Luna, Terra, and Sol on Amazon Bedrock with GPT-5.4 mini and nano on the OpenAI API. Instead of ranking models by dollars per million tokens, the harness measures cost per correct answer, cost per passing multi-turn research result, and cost per rubric-accepted professional deliverable. It uses one Responses API client and records timestamped result files, frozen grading prompts, and chart inputs. The post also documents important limits: sample sizes range from 48 to 198 items, Bedrock models ran with reasoning disabled, and API baselines used their defaults. AWS describes this as a practical configuration comparison rather than an estimate of intrinsic capability. Teams should rerun the method on 50 to 100 of their own tasks and include failures, retries, turns, latency, review, and rework in the decision.
Direct answer
AWS published an open-source benchmarking approach on September 11 for comparing GPT-5.6 Luna, Terra, and Sol on Amazon Bedrock with GPT-5.4 mini and nano on the OpenAI API. Instead of ranking models by dollars per million tokens, the harness measures cost per correct answer, cost per passing multi-turn research result, and cost per rubric-accepted professional deliverable. It uses one Responses API client and records timestamped result files, frozen grading prompts, and chart inputs. The post also documents important limits: sample sizes range from 48 to 198 items, Bedrock models ran with reasoning disabled, and API baselines used their defaults. AWS describes this as a practical configuration comparison rather than an estimate of intrinsic capability. Teams should rerun the method on 50 to 100 of their own tasks and include failures, retries, turns, latency, review, and rework in the decision.
Verified facts
The open-source harness uses one Responses API client to compare five models across single-call accuracy, multi-turn web-research trajectories, and rubric-graded professional deliverables.
AWS reports samples of 48 to 198 items. Bedrock GPT-5.6 models ran with reasoning disabled, while OpenAI API baselines used defaults; the results compare configurations, not intrinsic capability.
The method divides spending on both successful and failed attempts by successes, while recording agent turns, cumulative input tokens, latency, grading-prompt hashes, and timestamped result files.
What changed
- The cost unit shifts from token price to an outcome that clears a quality gate
- Agent turns and repeatedly transmitted context enter the total cost
- Failed attempts, review, and rework are included in model selection
- Model, region, price, and evaluation settings are preserved for reproducibility
Impact for AI users
The same nominal token price can produce very different success rates and turn counts, so a cheaper model may not deliver the lowest business cost. The methodology also warns against treating small gaps in a vendor sample as definitive: task distribution, reasoning settings, regional latency, pricing date, and the grader can all change the ranking. Before procurement or migration, teams should define an accepted outcome and then calculate the full cost using real failures and human rework.
Operating checklist
- Select 50 to 100 real tasks with known-good answers or explicit rubrics, then freeze the dataset and pass threshold.
- For every candidate, record the model snapshot, provider, region, reasoning settings, price date, output cap, and tool versions.
- Measure success rate, cost per accepted outcome, agent turns, tail latency, human review, and rework time together.
- Preserve raw responses, grading evidence, and failure categories, then rerun the same protocol when prices, models, or workload mix change.
AI frontier news and analysis, AI software and model tools, AI skill tutorials and validation methods, and AI account and permission guidance
FAQ
Why is token price insufficient for model selection?
Error rate, output length, and agent turns multiply total spend. A production workload buys accepted outcomes rather than isolated tokens.
Can the AWS results choose a model for every organization?
No. The authors explicitly recommend rerunning the harness on your own tasks and disclose sample-size, configuration, and timing limits.
Is the cross-platform result an intrinsic capability ranking?
No. Bedrock models ran with reasoning disabled while API baselines used defaults, so the post compares the tested deployment configurations.
Summary
An actionable model-cost evaluation defines an accepted outcome first, then accounts for failures, turns, latency, and rework. The vendor benchmark supplies a method and starting point; evidence from the team's own workload must determine the final choice.
This AI-assisted article is checked by ENHE AI automation for official sources, bilingual fields, media rights, page safety, and historical duplication before publication.
What this means for everyday users
The same nominal token price can produce very different success rates and turn counts, so a cheaper model may not deliver the lowest business cost. The methodology also warns against treating small gaps in a vendor sample as definitive: task distribution, reasoning settings, regional latency, pricing date, and the grader can all change the ranking. Before procurement or migration, teams should define an accepted outcome and then calculate the full cost using real failures and human rework.
Tools you may use
Related tutorials
[[zh]][[/zh]][[en]]Review a public website[[/en]]
SearchScope SEO/GEO
[[zh]][[/zh]][[en]]Access and usage guide[[/en]]
AI Prompt Management System | 418 Bilingual Prompts for Writing, SEO and AI Creation
High-Frequency AI Prompts for Work, Learning, and Teaching guide
High-Frequency AI Prompts for Work, Learning, and Teaching
Related Tools And Tutorials
Use the following ENHE AI sections to continue from the news signal into tool selection, account-service guidance, or practical learning.
Related reading
IskayPet Connects SharePoint to a Managed Knowledge Base: Validate Permissions Before Answers
AWS published IskayPet’s corporate knowledge assistant case study on October 22, 2026. The system connects Amazon Bedrock Managed Knowledge Bases to existing SharePoint repositories and answers cited questions across retail operations, veterinary care, onboarding, and procurement in four languages and three countries. This article turns the case into a practical acceptance plan for knowledge owners. It checks source scope, inherited permissions, hybrid retrieval, duplicate and fragment removal, citation quality, feedback, and token cost before expansion. It also separates the customer’s reported production results from what another organization should expect. A managed retrieval stack reduces infrastructure work, but it does not remove content ownership, access reviews, evaluation datasets, or human verification of important answers.
When Should an AI Agent Use a Local Small Model or the Cloud? A Hybrid Routing Checklist
AWS published an Adaptive Agentic Intelligence Framework guide using Strands Agents on October 21, 2026. The reference routes routine status checks and local vision work to small models through Ollama, while complex analysis goes to Amazon Bedrock. This durable guide helps teams decide when an AI agent should stay local and when it should use a cloud model. It classifies tasks, marks data that may leave the device, establishes local and cloud quality baselines, defines timeouts and fallbacks, applies one safety policy across both paths, and measures actual routing before estimating cost. The example can run on an 8 GB Raspberry Pi, but that hardware detail is not evidence that every workload belongs on a small local model.
Bayer Shares Production Lessons From 22 Specialist Agents: HTTP Success Is Not Task Success
AWS published Bayer Crop Science’s Agentic Decision Engine case study on October 23, 2026. The production system runs 22 specialist agents over governed analytics workflows, models, logs, and visualization tools. This article turns the architecture into five reusable acceptance checks for enterprise agent teams: persist the state needed to resume work, define identity and authorization at every boundary, scope each agent’s tool catalog, distinguish transport success from task success, and release to a gated group before broader access. It also explains why Bayer’s adoption and resolution figures are evidence about one implementation, not promises for another organization. The focus is operational proof, least privilege, rollback, and observable outcomes rather than prompt polish.
ChatGPT Slides Launches With Space Collaboration and PowerPoint Export
OpenAI added ChatGPT Slides on October 16, 2026, giving eligible Pro, Business, and Enterprise users with Space access a shared place to create and refine presentations. A deck can start from a conversation, uploaded files, an outline, or an example presentation that guides the visual direction. Users can edit text, images, shapes, and charts, invite teammates to work in real time, and leave shared comments. Finished work can be presented inside ChatGPT or exported on the web to PowerPoint, PDF, or Google Slides. This guide explains availability, a practical first-deck workflow, and the checks people should complete before sharing AI-assisted slides.
Nine-Stage AI Agent Evaluation Checklist: From First Commit to Production Drift
AWS published a nine-stage production AI-agent evaluation framework built around Amazon Bedrock AgentCore on October 16, 2026. It follows an agent from defining quality and comparing experiments through merge checks, deployment gates, canary scoring, adversarial testing, drift detection, live pulse monitoring, and feedback-driven improvement. The framework evaluates five dimensions—safety, quality, performance, cost, and operational control—at tool, individual-agent, and multi-agent levels. This durable guide converts the public framework into a vendor-neutral checklist for ordinary teams, with minimum evidence, ownership, and stop conditions designed to catch soft failures where latency and uptime remain normal while answer quality quietly declines over time.
OpenAI Updates Content Provenance: Extending the Work to EU Text
OpenAI's October 5 announcement sets out a phased approach to text watermarking, including API opt-in and a regional product rollout. The details are public and should be read alongside its broader media-provenance guidance. For publishers, the practical task is to distinguish which checks apply to text, images, and audio, and to keep evidence of how content was created and reviewed. A useful editorial record separates product coverage, source checks, permission to use materials, and AI disclosure. This article explains those distinctions and proposes a review workflow. A provenance signal is supporting evidence, not a substitute for factual verification or a conclusion about an author's identity or ownership.