Skip to content

Teach PR testing skill to validate VS Code extension changes#17963

Open
adamint wants to merge 2 commits into
microsoft:mainfrom
adamint:adamint/pr-testing-vscode-extension
Open

Teach PR testing skill to validate VS Code extension changes#17963
adamint wants to merge 2 commits into
microsoft:mainfrom
adamint:adamint/pr-testing-vscode-extension

Conversation

@adamint
Copy link
Copy Markdown
Member

@adamint adamint commented Jun 5, 2026

Description

This updates the PR testing skill so VS Code extension PRs are tested from the PR extension source or a VSIX built from that source. The dogfood CLI path is still used for product PRs, but extension-only PRs no longer get treated as blocked just because the CLI dogfood comment is missing or irrelevant to the extension changes.

The skill now calls out the extension-specific path: use a short checkout path, build with extension/build.sh, run corepack yarn run test, use focused corepack yarn run test:e2e for user-visible extension behavior, set ASPIRE_EXTENSION_E2E_CLI_PATH when testing with a specific CLI, and capture Extension Host/E2E diagnostics as evidence.

It also includes a concrete example report based on testing a VS Code extension PR, with artifact verification, build/unit validation, Extension Host E2E steps, diagnostics paths, and an overall result section.

Validation:

  • git diff --check
  • Used the updated flow while testing a real VS Code extension PR, including repo-built CLI validation and a focused Extension Host E2E run

Fixes # (issue)

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

Copilot AI review requested due to automatic review settings June 5, 2026 19:18
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 17963

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 17963"

adamint and others added 2 commits June 5, 2026 15:19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@adamint adamint force-pushed the adamint/pr-testing-vscode-extension branch from 694892b to 0e92a36 Compare June 5, 2026 19:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the pr-testing agent guidance to cover VS Code extension PR testing from PR source/VSIX, and it also includes a sizeable set of Aspire Dashboard changes (assistant focus restoration flows, text visualizer format picker changes, and log resource-prefix contrast updates) along with new/updated dashboard component tests.

Changes:

  • Expanded .agents/skills/pr-testing instructions to include VS Code extension-specific validation paths (source/VSIX, extension/build.*, yarn test, focused yarn test:e2e, evidence capture).
  • Added dashboard assistant focus-restore tracking via a new internal IAssistantDisplayContext implemented by AIContextProvider, and wired it through DI + layout/dialog components.
  • Updated dashboard UI/tests: switched TextVisualizerDialog format UI to FluentSelect, added WCAG contrast-driven resource-prefix styling and validation tests, and added accessibility state (aria-pressed) for assistant feedback buttons.
Show a summary per file
File Description
tests/Shared/TestDialogService.cs Implements additional dialog/panel async overloads used by newer dashboard tests.
tests/Shared/TestAIContextProvider.cs Extends test AI provider to implement new assistant display context surface for focus restoration tests.
tests/Aspire.Dashboard.Components.Tests/Shared/FluentUISetupHelpers.cs Registers TestAIContextProvider for both IAIContextProvider and IAssistantDisplayContext in test DI.
tests/Aspire.Dashboard.Components.Tests/Layout/MainLayoutTests.cs Adds focus-restore tests for header dialogs and assistant sidebar/modal transitions.
tests/Aspire.Dashboard.Components.Tests/Controls/TextVisualizerDialogTests.cs Updates tests for FluentSelect format picker and selection preservation across re-render.
tests/Aspire.Dashboard.Components.Tests/Controls/LogViewerTests.cs Adds contrast validation + style generation tests for resource prefix rendering.
tests/Aspire.Dashboard.Components.Tests/Controls/AssistantChatTests.cs Adds accessibility coverage for pressed-state on like/dislike buttons.
src/Aspire.Dashboard/wwwroot/css/app.css Adds --accent-*-text variables to support theme-specific foreground contrast for prefixes.
src/Aspire.Dashboard/Model/Assistant/IAssistantDisplayContext.cs Introduces internal assistant display context interface for focus/visibility coordination.
src/Aspire.Dashboard/Model/Assistant/AssistantDialogViewModel.cs Adds internal return-focus element id tracking for assistant dialogs.
src/Aspire.Dashboard/Model/Assistant/AIContextProvider.cs Implements IAssistantDisplayContext, adds focus target tracking, and routes modal dialog open via JS runtime for focus fixes.
src/Aspire.Dashboard/DashboardWebApplication.cs DI: registers AIContextProvider and exposes it via both assistant interfaces.
src/Aspire.Dashboard/Components/Layout/MainLayout.razor.cs Adds focus restoration plumbing for header dialogs and assistant sidebar hide/show lifecycle.
src/Aspire.Dashboard/Components/Layout/MainLayout.razor Assigns stable IDs to launch buttons and passes return-focus target into assistant sidebar dialog.
src/Aspire.Dashboard/Components/Dialogs/TextVisualizerDialog.razor.cs Updates selection handling to support FluentSelect and preserve selected option across parent re-renders.
src/Aspire.Dashboard/Components/Dialogs/TextVisualizerDialog.razor Replaces menu-button based format picker with FluentSelect.
src/Aspire.Dashboard/Components/Dialogs/AssistantSidebarDialog.razor.cs Uses IAssistantDisplayContext to control hide/expand behavior and focus target selection.
src/Aspire.Dashboard/Components/Dialogs/AssistantSidebarDialog.razor Updates close handler to async method.
src/Aspire.Dashboard/Components/Dialogs/AssistantModalDialog.razor.cs Adds optional JS-driven focus restoration on dialog close; routes sidebar switch via display context.
src/Aspire.Dashboard/Components/Controls/ResourcePrefixStyle.cs Centralizes resource-prefix inline style generation including theme-variable foreground selection.
src/Aspire.Dashboard/Components/Controls/LogViewer.razor Uses ResourcePrefixStyle for resource prefix styling instead of direct background-only styling.
src/Aspire.Dashboard/Components/Controls/AssistantChat.razor Adds aria-pressed state to like/dislike buttons for accessibility.
.agents/skills/pr-testing/SKILL.md Updates PR testing skill guidance to explicitly handle extension-only PR validation and evidence capture.

Copilot's findings

  • Files reviewed: 1/1 changed files
  • Comments generated: 1

Comment thread .agents/skills/pr-testing/SKILL.md
@adamint
Copy link
Copy Markdown
Member Author

adamint commented Jun 5, 2026

Tried this against an extension+CLI PR to make sure the new path gives useful output: #17955 (Stream AppHost discovery in VS Code extension).

Example run:

  • Checkout: /tmp/aspire-pr17955-test
  • PR head: f55c8fea2815616aedea587f2add3ee36acb1c15
  • Extension version: 1.13.0
  • extension/./build.sh: passed
  • corepack yarn run test: passed, 839 passing, 1 pending
  • Focused E2E:
ASPIRE_EXTENSION_E2E_SHARD=discovery-configuration \
ASPIRE_EXTENSION_E2E_SPEC=out/test-e2e/test-e2e/discoveryConfiguration.e2e.test.js \
ASPIRE_EXTENSION_E2E_CLI_PATH=/tmp/aspire-pr17955-test/artifacts/bin/Aspire.Cli/Debug/net10.0/aspire \
corepack yarn run test:e2e

That failed with a useful extension-specific signal:

Aspire workspace discovery and configuration E2E shows running workspace AppHosts while discovery is still pending
Running AppHost was not shown before slow discovery completed.

The last exported extension state had viewMode: "workspace", isWorkspaceAppHostDiscoveryComplete: false, and appHosts: [], while the AppHost output showed the apphost and dashboard were running. So the added guidance did send me to the right kind of validation for an extension PR.

A couple things I think we should fix in this PR before merging:

  1. The extension source/VSIX path is still under the same local/container execution-mode flow, but the repo-local PR container does not have the extension prerequisites. eng/scripts/aspire-pr-container/Dockerfile installs curl, docker.io, gh, git, libicu-dev, and unzip; extension/build.sh requires node, npm, VS Code/Insiders on PATH, and dotnet. I think the skill should explicitly say extension source/E2E validation runs on the host/local extension environment unless we add a dedicated extension-capable container.
  2. Step 4 and the critical reminders are still CLI-version centered. For extension source testing, the artifact check should be git rev-parse HEAD equals the PR headRefOid; for mixed CLI+extension PRs, the skill should say to verify/install the dogfood CLI and point ASPIRE_EXTENSION_E2E_CLI_PATH at that exact binary for E2E. Otherwise an agent can either stop on an extension-only PR because no CLI version matches, or accidentally test the extension against a stale local CLI.
  3. The E2E command should include the Linux/headless note from extension/CONTRIBUTING.md: run under xvfb-run -a when there is no desktop session. Otherwise Linux agents will fail before they test the PR behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants