Skip to main content
AI News & Updates

GitHub Copilot Evals Enters Preview: Regression-Test Skills, Agents, and MCP Changes

One good result does not prove a customization is stable

ENHE AI5 min3 views

GitHub launched Copilot Evals in public preview on November 17, 2026. It creates repeatable tests for skills, custom agents, and MCP tools and lets users compare runs before and after a customization change. This guide defines a minimum regression set: choose three realistic tasks, then specify the input, allowed tools, success criteria, and prohibited behavior for each. Preserve at least one baseline run and one post-change run, including the result, execution path, failed checks, and human review. Merge a customization only when the change preserves the original task, permission boundary, and evidence chain. A single passing answer is not enough because model output can vary, and an automated check cannot decide every domain-specific quality or safety question.

More topic tags (9)
AI NewsAuto PublishingAI ToolsSEOGitHub Copilot Evals技能测试AI Agents回归测试Vally
GitHub Copilot Evals Enters Preview: Regression-Test Skills, Agents, and MCP Changes

Key takeaways

Use three realistic tasks for the minimum regression set.
Assert result, process, permissions, and evidence.
Keep baseline and post-change runs, with human review of critical failures.

By ENHE AI · 2026-11-21 · AI-assisted drafting; pre-publication quality checks completed.

Step one: choose three realistic tasks

Do not start with “make the agent perform well.” Choose one common success path, one failure-prone boundary task, and one task involving tool permission, then freeze the input files or examples.

Each task needs an inspectable output, such as a changed file, a sourced report, or an explicit receipt that an external action was not taken. Vague quality impressions do not support stable comparison.

Step two: write four assertion groups

Result assertions check the final output; process assertions check tools and steps; permission assertions check account, data, and external-action boundaries; evidence assertions verify tests, sources, or files actually exist.

Do not assert completion from the agent's statement. Any state that needs a file, test, page, source, or human record must be checked independently.

Step three: save baseline and post-change runs

Run a baseline before changing the skill, agent, or MCP. Save version, model, input, output, and check details. After the change, rerun the identical task instead of changing the input at the same time.

Model output varies, so repeat critical tasks and inspect failure modes. This article does not prescribe a statistical sample size and did not run Vally or measure a configuration win rate.

A phone and computer represent baseline and post-change comparisons. Contextual stock photo, not evaluation scores, agent output, or test logs. Image: Unsplash.
A phone and computer represent baseline and post-change comparisons. Contextual stock photo, not evaluation scores, agent output, or test logs. Image: Unsplash.

Step four: decide whether to merge

Merge only when the new configuration preserves the original success path, fixes the target issue, does not broaden permission, and keeps the evidence chain intact. A new capability that breaks another task is a regression.

Automated checks cannot judge every business meaning. Keep domain or owner review for money, compliance, publication, and high-impact actions; a machine pass does not replace responsibility.

FAQ

Is one passing run enough?

No. GitHub notes that AI output can change, so reuse the same eval to inspect stability and regressions.

Can an eval replace human review?

No. It is useful for repeatable explicit checks, while business meaning, risk, and final responsibility still need human judgment.

Fact evidence

  • GitHub Copilot Evals entered public preview for Copilot Individual, Business, and Enterprise plans. — GitHub Changelog — GitHub Copilot Evals public preview
  • An eval is a repeatable test with a realistic task, a description of a good result, and checks for the result and the path taken. — GitHub Changelog — GitHub Copilot Evals public preview
  • Users approve each eval run before it starts and can compare runs before and after a customization change. — GitHub Changelog — GitHub Copilot Evals public preview
  • Copilot Evals uses the extensible local Vally evaluation tool as its engine. — GitHub Changelog — GitHub Copilot Evals public preview

Related tools and reading

These production pages support the same decision, workflow, or safety boundary.

Conclusion

Copilot Evals matters because skills, agents, and MCP changes can share the same inputs, assertions, and comparable receipts—not because it adds another score. Start with a small, realistic regression set.

What this means for everyday users

Merge only when the new configuration preserves the original success path, fixes the target issue, does not broaden permission, and keeps the evidence chain intact. A new capability that breaks another task is a regression.

Related reading

AWS Launches AgentCore Evaluations for Testing Any Agent Framework

AWS Launches AgentCore Evaluations for Testing Any Agent Framework. The official source dated August 2026 describes a concrete product, research, or governance change rather than a universal guarantee. This article separates what is available now from preview or planned access, then translates the change into one ordinary-user task: establishing repeatable offline evaluations, online monitoring, and human spot checks for an AI agent. Before using it, readers should verify account eligibility, workspace permissions, data boundaries, model or service cost, human review, audit logs, and rollback. A small reversible pilot with explicit acceptance checks is safer than copying a headline result or assuming that a new integration can publish, merge, or make decisions without approval. The source set is linked so teams can recheck availability and scope when the product changes.

How to Build an AI Agent Evaluation Baseline: From Offline Tests to Production Review

How to Build an AI Agent Evaluation Baseline: From Offline Tests to Production Review. The official source dated August 2026 describes a concrete product, research, or governance change rather than a universal guarantee. This article separates what is available now from preview or planned access, then translates the change into one ordinary-user task: establishing a repeatable baseline for AI-agent quality, risk, cost, and human review. Before using it, readers should verify account eligibility, workspace permissions, data boundaries, model or service cost, human review, audit logs, and rollback. A small reversible pilot with explicit acceptance checks is safer than copying a headline result or assuming that a new integration can publish, merge, or make decisions without approval. The source set is linked so teams can recheck availability and scope when the product changes.

Nine-Stage AI Agent Evaluation Checklist: From First Commit to Production Drift

AWS published a nine-stage production AI-agent evaluation framework built around Amazon Bedrock AgentCore on October 16, 2026. It follows an agent from defining quality and comparing experiments through merge checks, deployment gates, canary scoring, adversarial testing, drift detection, live pulse monitoring, and feedback-driven improvement. The framework evaluates five dimensions—safety, quality, performance, cost, and operational control—at tool, individual-agent, and multi-agent levels. This durable guide converts the public framework into a vendor-neutral checklist for ordinary teams, with minimum evidence, ownership, and stop conditions designed to catch soft failures where latency and uptime remain normal while answer quality quietly declines over time.

From Chat Boxes to Personal AI Companions: AI Assistants Are Entering the Desktop Execution Era

AI assistants are moving from answering questions toward continuing real tasks. AI agents, MCP tool ecosystems, personal memory, and local workbenches are pushing this shift together. For users, the real value is not another chat box, but less repeated context setup and more continuity from thinking to doing.

ChatGPT Live Dictation Launches: Review Transcription, Structure, and Privacy for Long Prompts

OpenAI introduced ChatGPT Live Dictation on November 18, 2026. Speech now appears as text while the user talks, the live transcript can be shown or hidden, and the former ten-minute recording limit has been removed across a global rollout to all plans and major platforms. This guide turns the update into a practical long-prompt check. Say four labels aloud—goal, context, constraints, and output format—correct names and numbers while the text is visible, then review omissions, substitutions, and sensitive details before sending. It also separates editable dictation from a live voice conversation and points users to audio retention and sharing controls. Real-time text is useful feedback, not proof that the transcript is complete or accurate.

GitHub Copilot Browser Use Enters Preview: Test Signed-In Sessions and Permissions Read-Only First

GitHub released Browser Use in public preview on November 17, 2026 for Copilot CLI, the Copilot app, VS Code, and JetBrains, paired with the GitHub Copilot Chrome extension. This guide starts with a read-only acceptance path. Use a test account and a fixed page, confirm that the extension reports Connected, record the tab scope and site permissions, and compare the elements and screenshots the agent uses with the expected page. Then verify that sign-in, form submission, deletion, and external communication stop for explicit approval. Expand only after the read-only flow is repeatable, permissions can be revoked, and failures do not cause unintended actions. The article does not claim a hands-on browser-use test or permission behavior beyond GitHub's documentation.