GitHub Copilot Evals Enters Preview: Regression-Test Skills, Agents, and MCP Changes
One good result does not prove a customization is stable
GitHub launched Copilot Evals in public preview on November 17, 2026. It creates repeatable tests for skills, custom agents, and MCP tools and lets users compare runs before and after a customization change. This guide defines a minimum regression set: choose three realistic tasks, then specify the input, allowed tools, success criteria, and prohibited behavior for each. Preserve at least one baseline run and one post-change run, including the result, execution path, failed checks, and human review. Merge a customization only when the change preserves the original task, permission boundary, and evidence chain. A single passing answer is not enough because model output can vary, and an automated check cannot decide every domain-specific quality or safety question.
Key takeaways
By ENHE AI · 2026-11-21 · AI-assisted drafting; pre-publication quality checks completed.
Step one: choose three realistic tasks
Do not start with “make the agent perform well.” Choose one common success path, one failure-prone boundary task, and one task involving tool permission, then freeze the input files or examples.
Each task needs an inspectable output, such as a changed file, a sourced report, or an explicit receipt that an external action was not taken. Vague quality impressions do not support stable comparison.
Step two: write four assertion groups
Result assertions check the final output; process assertions check tools and steps; permission assertions check account, data, and external-action boundaries; evidence assertions verify tests, sources, or files actually exist.
Do not assert completion from the agent's statement. Any state that needs a file, test, page, source, or human record must be checked independently.
Step three: save baseline and post-change runs
Run a baseline before changing the skill, agent, or MCP. Save version, model, input, output, and check details. After the change, rerun the identical task instead of changing the input at the same time.
Model output varies, so repeat critical tasks and inspect failure modes. This article does not prescribe a statistical sample size and did not run Vally or measure a configuration win rate.
Step four: decide whether to merge
Merge only when the new configuration preserves the original success path, fixes the target issue, does not broaden permission, and keeps the evidence chain intact. A new capability that breaks another task is a regression.
Automated checks cannot judge every business meaning. Keep domain or owner review for money, compliance, publication, and high-impact actions; a machine pass does not replace responsibility.
FAQ
Is one passing run enough?
No. GitHub notes that AI output can change, so reuse the same eval to inspect stability and regressions.
Can an eval replace human review?
No. It is useful for repeatable explicit checks, while business meaning, risk, and final responsibility still need human judgment.
Fact evidence
- GitHub Copilot Evals entered public preview for Copilot Individual, Business, and Enterprise plans. — GitHub Changelog — GitHub Copilot Evals public preview
- An eval is a repeatable test with a realistic task, a description of a good result, and checks for the result and the path taken. — GitHub Changelog — GitHub Copilot Evals public preview
- Users approve each eval run before it starts and can compare runs before and after a customization change. — GitHub Changelog — GitHub Copilot Evals public preview
- Copilot Evals uses the extensible local Vally evaluation tool as its engine. — GitHub Changelog — GitHub Copilot Evals public preview
Related tools and reading
These production pages support the same decision, workflow, or safety boundary.
Conclusion
Copilot Evals matters because skills, agents, and MCP changes can share the same inputs, assertions, and comparable receipts—not because it adds another score. Start with a small, realistic regression set.
What this means for everyday users
Merge only when the new configuration preserves the original success path, fixes the target issue, does not broaden permission, and keeps the evidence chain intact. A new capability that breaks another task is a regression.
Related reading
AWS Launches AgentCore Evaluations for Testing Any Agent Framework
AWS Launches AgentCore Evaluations for Testing Any Agent Framework. The official source dated August 2026 describes a concrete product, research, or governance change rather than a universal guarantee. This article separates what is available now from preview or planned access, then translates the change into one ordinary-user task: establishing repeatable offline evaluations, online monitoring, and human spot checks for an AI agent. Before using it, readers should verify account eligibility, workspace permissions, data boundaries, model or service cost, human review, audit logs, and rollback. A small reversible pilot with explicit acceptance checks is safer than copying a headline result or assuming that a new integration can publish, merge, or make decisions without approval. The source set is linked so teams can recheck availability and scope when the product changes.
How to Build an AI Agent Evaluation Baseline: From Offline Tests to Production Review
How to Build an AI Agent Evaluation Baseline: From Offline Tests to Production Review. The official source dated August 2026 describes a concrete product, research, or governance change rather than a universal guarantee. This article separates what is available now from preview or planned access, then translates the change into one ordinary-user task: establishing a repeatable baseline for AI-agent quality, risk, cost, and human review. Before using it, readers should verify account eligibility, workspace permissions, data boundaries, model or service cost, human review, audit logs, and rollback. A small reversible pilot with explicit acceptance checks is safer than copying a headline result or assuming that a new integration can publish, merge, or make decisions without approval. The source set is linked so teams can recheck availability and scope when the product changes.
Nine-Stage AI Agent Evaluation Checklist: From First Commit to Production Drift
AWS published a nine-stage production AI-agent evaluation framework built around Amazon Bedrock AgentCore on October 16, 2026. It follows an agent from defining quality and comparing experiments through merge checks, deployment gates, canary scoring, adversarial testing, drift detection, live pulse monitoring, and feedback-driven improvement. The framework evaluates five dimensions—safety, quality, performance, cost, and operational control—at tool, individual-agent, and multi-agent levels. This durable guide converts the public framework into a vendor-neutral checklist for ordinary teams, with minimum evidence, ownership, and stop conditions designed to catch soft failures where latency and uptime remain normal while answer quality quietly declines over time.
From Chat Boxes to Personal AI Companions: AI Assistants Are Entering the Desktop Execution Era
AI assistants are moving from answering questions toward continuing real tasks. AI agents, MCP tool ecosystems, personal memory, and local workbenches are pushing this shift together. For users, the real value is not another chat box, but less repeated context setup and more continuity from thinking to doing.
ChatGPT Live Dictation Launches: Review Transcription, Structure, and Privacy for Long Prompts
OpenAI introduced ChatGPT Live Dictation on November 18, 2026. Speech now appears as text while the user talks, the live transcript can be shown or hidden, and the former ten-minute recording limit has been removed across a global rollout to all plans and major platforms. This guide turns the update into a practical long-prompt check. Say four labels aloud—goal, context, constraints, and output format—correct names and numbers while the text is visible, then review omissions, substitutions, and sensitive details before sending. It also separates editable dictation from a live voice conversation and points users to audio retention and sharing controls. Real-time text is useful feedback, not proof that the transcript is complete or accurate.
GitHub Copilot Browser Use Enters Preview: Test Signed-In Sessions and Permissions Read-Only First
GitHub released Browser Use in public preview on November 17, 2026 for Copilot CLI, the Copilot app, VS Code, and JetBrains, paired with the GitHub Copilot Chrome extension. This guide starts with a read-only acceptance path. Use a test account and a fixed page, confirm that the extension reports Connected, record the tab scope and site permissions, and compare the elements and screenshots the agent uses with the expected page. Then verify that sign-in, form submission, deletion, and external communication stop for explicit approval. Expand only after the read-only flow is repeatable, permissions can be revoked, and failures do not cause unintended actions. The article does not claim a hands-on browser-use test or permission behavior beyond GitHub's documentation.
