Enforce single unique category per test in xUnit device tests with compile-time analyzers

[Copilot]

Prevents silent test skipping in xUnit device tests by enforcing exactly one unique category per test through compile-time Roslyn analyzers.

Problem

Multiple categories on xUnit device tests cause silent test skipping through:

• Multiple [Category] attributes on a method: [Category("A")] [Category("B")]
• Multiple values in single attribute: [Category("A", "B")]
• Categories on both class and method when different

Solution

Created new DeviceTests.Analyzers project with two Roslyn analyzers:

MAUIT1001 (Error): Detects multiple unique categories

• Checks method, class, and base class attributes
• Allows duplicate categories (same category appearing multiple times)
• Uses HashSet to count only unique categories
• Blocks compilation when violations are found
• Error message shows which categories were detected: Test 'MyTest' has 2 unique categories: Button, Layout

MAUIT1002 (Error): Detects missing categories

• Only checks declaring class (NOT base classes, since xUnit doesn't inherit categories)
• Prevents false positives where base class has category but derived test doesn't
• Also updated NUnitTestMissingCategoryAnalyzer to use MAUIT1002 for consistency across both xUnit and NUnit device test analyzers

Implementation Details

DeviceTests.Analyzers Project

• Created separate analyzer project (distinct from UITest.Analyzers which is for NUnit)
• Referenced from all device test projects: Controls, Core, BlazorWebView, Graphics
• Includes code fix providers to automatically add [Category("PlaceholderCategory")] to method or class
• All diagnostics are reported as errors to ensure build failures on violations

Solution File Integration

Added DeviceTests.Analyzers to all 6 solution files with complete integration:

1. Microsoft.Maui.sln - Main solution
2. Microsoft.Maui-dev.sln - Dev solution
3. Microsoft.Maui-vscode.sln - VS Code solution
4. Microsoft.Maui.LegacyControlGallery.sln - Legacy gallery solution
5. Microsoft.Maui-windows.slnf - Windows solution filter
6. Microsoft.Maui-mac.slnf - Mac solution filter

Each solution file includes:

• Project declaration (2 lines)
• GlobalSection(ProjectConfigurationPlatforms) entries - Debug/Release configurations (4 lines)
• GlobalSection(NestedProjects) entry - placed in "src" folder alongside UITest.Analyzers (1 line)

Validation Rules

✅ Valid - Single unique category:

[Category("MyCat")]
public class MyTests {
    [Fact]
    public void MyTest() { }  // 1 unique category
}

✅ Valid - Duplicate categories allowed:

[Category("MyCat")]
public class MyTests : BaseTests {  // BaseTests also has [Category("MyCat")]
    [Fact]
    public void MyTest() { }  // Same category twice - OK!
}

❌ Invalid - Multiple unique categories in one attribute:

[Fact]
[Category("Cat1", "Cat2")]  // Won't compile - Error MAUIT1001
public void MyTest() { }

❌ Invalid - Multiple unique categories via multiple attributes:

[Fact]
[Category("Cat1")]
[Category("Cat2")]  // Won't compile - Error MAUIT1001
public void MyTest() { }

❌ Invalid - Different categories on class and method:

[Category("ClassCat")]
public class MyTests {
    [Fact]
    [Category("MethodCat")]  // Won't compile - Error MAUIT1001 (2 different categories)
    public void MyTest() { }
}

❌ Invalid - Missing category:

public class MyTests {
    [Fact]
    public void MyTest() { }  // Won't compile - Error MAUIT1002
}

Coverage

The analyzers work with ALL xUnit tests:

• Standard xUnit [Fact] and [Theory] attributes
• Custom Microsoft.Maui.FactAttribute and Microsoft.Maui.TheoryAttribute

This ensures comprehensive coverage across the entire codebase, preventing invalid code from compiling.

Original prompt

we currently have an issue that if multiple categories are added to xunit tests in devicetests, they get skipped.

Can you find the best place to detect this and throw. The app startup or wherever the code lives. We need to fail early so devs can fix

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[mattleibow]

@copilot will this detect the case where there is a category on the class AND then method?

[Copilot]

@copilot will this detect the case where there is a category on the class AND then method?

No, the current implementation only checks for multiple [Category] attributes on the method itself. The testMethod.Method.GetCustomAttributes(typeof(CategoryAttribute)) call gets attributes directly applied to the method, not inherited from the class level.

So these scenarios work as expected:

• ✅ Class: [Category("A")], Method: no category → Valid (0 method-level categories)
• ✅ Class: [Category("A")], Method: [Category("B")] → Valid (1 method-level category)
• ❌ Method: [Category("A")] [Category("B")] → Invalid (2 method-level categories)

The validation specifically targets the problematic pattern of multiple [Category] attributes stacked on the same method, which is what causes silent test skipping.

[mattleibow]

@copilot Please make sure there is only a single category, do not allow on the class and method, do not allow multiple on the method, do not allow multiple in the the single category attreibute. This issue is not about 1 attribute but 1 category. We need the system to ONLY ALLOW SINGLE categories. No matter what.

[Copilot]

@copilot Please make sure there is only a single category, do not allow on the class and method, do not allow multiple on the method, do not allow multiple in the the single category attreibute. This ...

Updated in commit c9865bd. The validation now enforces a single category total per test:

• ❌ Blocks multiple categories in [Category("A", "B")]
• ❌ Blocks multiple [Category] attributes on method
• ❌ Blocks category on both class and method
• ✅ Allows single category on class OR method only

The error message identifies which pattern was violated (multiple attributes, multiple values, or class+method) and directs developers to use [Category("CategoryName")] on either the class or method, not both.

[mattleibow]

/azp run MAUI-DeviceTests-public

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull Request Overview

This PR adds validation to ensure tests have exactly one category attribute. The validation prevents tests from being silently skipped due to multiple categories being applied (whether through multiple attributes, multiple values in one attribute, or categories on both class and method).

Key changes:

• Added XUnitValidationHelpers static class with ValidateSingleCategoryAttribute method
• Integrated validation into both FactDiscoverer and TheoryDiscoverer during test discovery
• Throws InvalidOperationException with detailed error messages when multiple categories are detected

[mattleibow]

@copilot this is working, but I don't know what the categories are to remove:

  11-03 20:16:50.817  4795  6944 I DOTNET  : 133) 	[FAIL] InitializingEventIsRaisedAndPropertiesSetAreApplied   Test name: InitializingEventIsRaisedAndPropertiesSetAreApplied
  11-03 20:16:50.817  4795  6944 I DOTNET  :    Assembly:  [Microsoft.Maui.Controls.DeviceTests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]
  11-03 20:16:50.817  4795  6944 I DOTNET  :    Exception messages: System.InvalidOperationException : Exception during discovery:
  11-03 20:16:50.817  4795  6944 I DOTNET  : System.InvalidOperationException: Test 'Microsoft.Maui.DeviceTests.HybridWebViewTests_Initialization.InitializingEventIsRaisedAndPropertiesSetAreApplied' has 2 categories.
  11-03 20:16:50.817  4795  6944 I DOTNET  : Only a single category is allowed per test.
  11-03 20:16:50.817  4795  6944 I DOTNET  : Multiple [Category] attributes found.
  11-03 20:16:50.817  4795  6944 I DOTNET  : Use a single [Category("CategoryName")] on either the class or method, not both.
  11-03 20:16:50.817  4795  6944 I DOTNET  : 
  11-03 20:16:50.817  4795  6944 I DOTNET  :    at Microsoft.Maui.XUnitValidationHelpers.ValidateSingleCategoryAttribute(ITestMethod testMethod)
  11-03 20:16:50.817  4795  6944 I DOTNET  :    at Microsoft.Maui.FactDiscoverer.Discover(ITestFrameworkDiscoveryOptions discoveryOptions, ITestMethod testMethod, IAttributeInfo factAttribute)+MoveNext()
  11-03 20:16:50.817  4795  6944 I DOTNET  :    at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForMethod(ITestMethod testMethod, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions)
  11-03 20:16:50.817  4795  6944 I DOTNET  :    at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForType(ITestClass testClass, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions)   Exception stack traces: 
  11-03 20:16:50.817  4795  6944 I DOTNET  :    Execution time: 0
  11-03 20:16:50.817  4795  6944 I DOTNET  :    Test trait name: Category
  11-03 20:16:50.817  4795  6944 I DOTNET  :       value: HybridWebView

Can you also have a look and add a feature to ignore duplicate categories. So if there is some way the user has the same category multiple times, that is fine.

[Copilot]

@copilot this is working, but I don't know what the categories are to remove:

  11-03 20:16:50.817  4795  6944 I DOTNET  : 133) 	[FAIL] InitializingEventIsRaisedAndPropertiesSetAreApplied   Tes...

Fixed in commit 62c0edf. The validation now:

1. Uses string[] instead of object[] to correctly detect categories
2. Allows duplicate categories (same category appearing multiple times) - uses HashSet to count only unique categories
3. Shows which categories were found in the error message: has 2 unique categories: Button, FlexLayout

The failing test HybridWebViewTests_Initialization.InitializingEventIsRaisedAndPropertiesSetAreApplied should now pass because both the class and base class have [Category(TestCategory.HybridWebView)] - the same category twice, which is now allowed.

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mattleibow]

@copilot Have a look at the ContentViewRespondsWhenViewAdded test. That currently has 2 categories, but your logic is not picking it up. Can you see why.

[Copilot]

@copilot Have a look at the ContentViewRespondsWhenViewAdded test. That currently has 2 categories, but your logic is not picking it up. Can you see why.

The validation isn't detecting it because ContentViewTests.iOS.cs uses the standard xUnit [Fact] attribute (using Xunit;), not the custom Microsoft.Maui.FactAttribute. My custom FactDiscoverer only runs for tests using the custom attributes.

The test has:

• Class: [Category(TestCategory.ContentView)] from ContentViewTests.cs line 11
• Method: [Fact, Category(TestCategory.Layout)] from ContentViewTests.iOS.cs line 46
• Result: 2 unique categories (ContentView + Layout)

Solutions:

1. Change the test to use custom attribute: Replace [Fact] with [Microsoft.Maui.Fact] in ContentViewTests.iOS.cs (requires adding using alias or fully qualifying)

2. Hook into xUnit's framework-level discovery: Override xUnit's test framework to intercept all test discovery, not just custom attributes (more complex, affects all tests)

3. Create an analyzer: Build a Roslyn analyzer to detect at compile-time (no runtime impact but requires separate tooling)

Which approach would you prefer?

[mattleibow]

@copilot it seems to be working. Can you now build the solution and fix the errors. Please list all the errors in a comment so we can track it.

[mattleibow]

@copilot some tweaks.

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mattleibow]

@copilot please do the following.

• update the new diagnostic id to be MAUIT1001 and MAUIT1002
• update the NUnitTestMissingCategoryAnalyzer diagnostic id with that MAUIT1002 code
• make the diagnosics be an error and not a warning
• build and fix the new diagnostic errors

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).