Add auto-unquarantine tests agent and workflow

[Copilot]

Description

Implements automated weekly unquarantining of stable tests based on CI quarantine reports. Tests with zero failures across all platforms (linux, macos, windows) and sufficient run coverage (≥90 runs per OS) are automatically unquarantined via batch PR.

Agent Definition (.github/agents/auto-unquarantine-tests.md)

• Defines execution logic for issue discovery, report parsing, and test unquarantining
• Parses OS-specific failure counts from quarantine report tables in issue bodies
• Extracts fully-qualified test method names from issue titles
• Invokes QuarantineTools to remove [QuarantinedTest] attributes
• Includes error handling for partial failures and missing test methods
• Contains complete instructions for fetching issues, analyzing reports, unquarantining tests, building, and creating PRs

Workflow Implementation (.github/workflows/auto-unquarantine-tests.yml)

• Schedule: Weekly (Monday 2 AM UTC) + manual dispatch
• Architecture: Workflow invokes the agent by creating an issue and assigning it to copilot-swe-agent
• Pipeline:
  1. Read agent instructions from .github/agents/auto-unquarantine-tests.md
  2. Create an issue with the agent instructions and configuration parameters
  3. Assign the issue to copilot-swe-agent
  4. The agent executes all work (issue discovery, report parsing, unquarantining, building, PR creation)
• Configuration: min_runs_threshold (default: 90) passed to agent via issue body
• Simplicity: Workflow reduced from 483 lines to 104 lines by delegating execution to the agent

Example Usage

Manual trigger with custom threshold:

gh workflow run auto-unquarantine-tests.yml \
  -f min_runs_threshold=100

Quarantine report format expected:

| OS | Last 100 runs |
| --- | ------------- |
| linux (0/100) | ✅ |
| macos (0/95) | ✅ |
| windows (0/98) | ✅ |

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
  • No
• Does the change require an update in our Aspire docs?
  • Yes
  • No

Original prompt

Add Auto-Unquarantine Tests Agent and Workflow

Summary

Add a new Copilot agent and GitHub workflow to automatically unquarantine stable tests that have had zero failures in recent CI runs.

Files to Create

1. .github/agents/auto-unquarantine-tests.md

Create the agent description file with the following content:

---
name: auto-unquarantine-tests
description: Automatically unquarantines stable tests that have had zero failures in recent CI runs
tools: ["bash", "view", "edit"]
---

You are a specialized test management agent for the dotnet/aspire repository. Your primary function is to automatically identify and unquarantine tests that have been stable (zero failures) across all platforms in recent CI runs.

## Overview

This agent runs on a weekly schedule to:
1. Find all open issues with the `quarantined-test` label
2. Analyze the "Report for Quarantined runs" in each issue's description
3. Identify tests that have had 0 failures across all OSes (linux, macos, windows) with sufficient run coverage
4. Unquarantine those tests using the `tools/QuarantineTools` utility
5. Create a single PR that closes all related issues

## Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `MIN_RUNS_THRESHOLD` | 90 | Minimum number of test runs required per OS to consider the test stable |
| `TARGET_BRANCH` | main | Branch to target for the PR |
| `REQUIRED_OSES` | linux, macos, windows | All OSes that must have zero failures |

## Task Execution Steps

### 1. Fetch Quarantined Test Issues

Use the GitHub API to fetch all open issues with the `quarantined-test` label:

```bash
gh issue list --repo dotnet/aspire --label "quarantined-test" --state open --json number,title,body --limit 100

2. Parse Issue Bodies for Quarantine Reports

For each issue, look for the quarantine report section in the issue body. The report format is:

<!-- BEGIN: Github workflow runs test report -->

## Report for Quarantined runs

### Recent runs:

| OS | Last 100 runs |
| --- | ------------- |
| linux (X/100) | ... |
| macos (X/100) | ... |
| windows (X/100) | ... |

Parsing rules:

• Extract the failure count from the pattern (X/Y) where X is failures and Y is total runs
• The test is stable if:
  • All three OSes (linux, macos, windows) have entries
  • All failure counts (X) are 0
  • All run counts (Y) are >= MIN_RUNS_THRESHOLD (default: 90)

Alternative "No failures" format:
If an OS shows No failures ✅ without a specific count, treat it as 0 failures but verify the run count from the summary table.

3. Extract Test Method Names

From stable issues, extract the fully-qualified test method name from:

1. The issue title (usually contains the test name)
2. The issue body (often has the full namespace.class.method)

Common title patterns:

• Failing test: Namespace.Class.Method
• Flaky test: Namespace.Class.Method
• [Failing test]: Method

If only the method name is available, search for the fully-qualified name:

grep -r "public.*void.*MethodName\|public.*async.*Task.*MethodName" tests/ --include="*.cs" -l

Then examine the file to get the full namespace and class name.

4. Unquarantine Tests Using QuarantineTools

For each stable test, run QuarantineTools to remove the [QuarantinedTest] attribute:

dotnet run --project tools/QuarantineTools -- -u <Namespace.Type.Method>

Process all tests before building:

• Run QuarantineTools for each test
• Track successes and failures
• Continue processing even if some tests fail

5. Build and Verify

After all QuarantineTools invocations, build the affected test projects:

# Build all affected test projects
dotnet build tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj
dotnet build tests/Aspire.Hosting.Tests/Aspire.Hosting.Tests.csproj
# ... other affected projects

If build fails:

• Report the specific compilation error
• Note which tests could not be unquarantined
• Continue with tests that built successfully

6. Create Commit and Pull Request

Commit message format:

Unquarantine stable tests

The following tests have had 0 failures in the last 100+ runs across all platforms
and are being unquarantined:

- Unquarantined: TestMethod1
- Unquarantined: TestMethod2
- ...

These tests were automatically identified as stable by the auto-unquarantine-tests agent.

PR Title:

Unquarantine {N} stable tests with zero failures

PR Description:

## Summary

This PR automatically unquarantines tests that have been stable (zero failures) across all platforms in recent CI runs.

### Stability Criteria
- Minimum runs per OS: {MIN_RUNS_THRESHOLD}
- Required OSes: linux, macos, windows
- Required failure count: 0 on all OSes

## Unquarantined Tests

| Test Method | File | Issue | Runs (linux) | Runs (macos) | Runs (windows) |
|-------------|------|-------|--------------|--------------|----------------|
| TestMethod1...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created from Copilot chat.*
>

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13775

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13775"

[radical]

@copilot This workflow should essentially just invoke the auto-quarantine-tests agent. And all the real work should be done by the agent.

[Copilot]

Done. The workflow now simply creates an issue with the agent instructions and assigns it to copilot-swe-agent. All the actual work (fetching issues, parsing reports, unquarantining tests, building, and creating PRs) is now handled by the agent.

The workflow is reduced from 483 lines to 104 lines and now just:

1. Reads the agent instructions from .github/agents/auto-unquarantine-tests.md
2. Creates an issue with those instructions and configuration parameters
3. Assigns the issue to copilot-swe-agent

Commit: d74e22d

[radical]

@copilot review the changes