Skip to content

[ci] Run xharness DeviceTests on dnceng-public Helix #30544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 108 commits into
base: main
Choose a base branch
from

Conversation

rmarinho
Copy link
Member

@rmarinho rmarinho commented Jul 10, 2025

Description of Change

Enable a way to run device tests on Helix with xharness.
Right now we going to support both modes, helix and non helix as we are doing the move.

About helix

Check available queues https://helix.dot.net (right now we just going to use open)

Documentation about xharness on Helix https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/Readme.md#android-apk-payloads

Run locally from the Mac

The arcade system uses ./build.sh and /build.cmd as the start point, and it will look at the Build.props file for information of what projects to build

#build the Build tasks
./build.sh -restore -build -configuration Release -projects '$(Build.SourcesDirectory)/Microsoft.Maui.BuildTasks.slnf' /bl:BuildBuildTasks.binlog -warnAsError false
 
 #build the devicetests
./build.sh -restore -build -configuration Release /p:BuildDeviceTests=true /bl:BuildDeviceTests.binlog -warnAsError false

#send to helix
./eng/common/msbuild.sh /Users/ruimarinho/dotnet/maui/eng/helix_xharness.proj /restore /p:TreatWarningsAsErrors=false /t:Test /p:TargetOS=android /bl:sendhelix.binlog -verbosity:diag 

This pull request introduces several updates to streamline device test configurations and address project path inconsistencies across multiple solution files. Key changes include modifying solution file references for the MauiBlazorWebView.DeviceTests project, enhancing build and test pipelines for device tests, and adding support for Helix and XHarness testing.

Updates to solution files:

  • Adjusted the project path for MauiBlazorWebView.DeviceTests in multiple solution files (Microsoft.Maui-dev.sln, Microsoft.Maui-mac.slnf, Microsoft.Maui-vscode.sln, Microsoft.Maui-windows.slnf, Microsoft.Maui.LegacyControlGallery.sln, and Microsoft.Maui.sln) to use src\BlazorWebView\tests\DeviceTests\MauiBlazorWebView.DeviceTests.csproj instead of src\BlazorWebView\tests\MauiDeviceTests\MauiBlazorWebView.DeviceTests.csproj. [1] [2] [3] [4] [5] [6]

Enhancements to build and test pipelines:

  • Updated eng/Build.props to conditionally include device test projects in the build process when BuildDeviceTests is set to true. This includes projects for Controls, Core, Graphics, Essentials, and MauiBlazorWebView device tests.
  • Added a new project file, eng/helix_xharness.proj, to support Helix and XHarness testing for iOS and Android platforms. This includes configurations for test scenarios, payload directories, and handling app bundles and APKs.
  • Introduced a new pipeline template, eng/pipelines/arcade/stage-device-tests.yml, for running device tests on iOS and Android using Helix. This includes parameters for pool configurations, build settings, and artifact publishing.

Pipeline variable and configuration updates:

  • Modified eng/pipelines/arcade/variables.yml to update the _buildScriptMacOS variable, changing -warnAsError 0 to -warnAsError false.
  • Updated eng/pipelines/common/variables.yml to include a new condition for the maui-pr-devicetests build definition.
  • Added a new variable template to eng/pipelines/device-tests.yml for pool providers.

These changes collectively enhance the project's ability to manage device tests more efficiently while ensuring consistency in project paths across solution files.

Issues Fixed

Fixes #30466

@@ -3,8 +3,8 @@ namespace Microsoft.Maui.TestUtils.DeviceTests.Runners
{
public class HeadlessRunnerOptions
{
public string TestResultsFilename { get; set; } = "TestResults.xml";
public string TestResultsFilename { get; set; } = "testResults.xml";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Helix looks at this to grab the results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Run device tests on dnceng/helix
2 participants