We have two open issues right now related to how tests run in parallel:

• Feature Request - Test case parallelization when there is no class/collection fixture #1986
• Feature Request: Run Tests in Parallel with Shared Context #2055

As I work through what the options for parallelism should be, I'm thinking maybe trying to set up a strict list of choices is probably the wrong strategy, so here's what I'm considering.

For starters, we should state where we are today: we have two modes of parallelization, with some modifiers.

• The first is "nothing runs in parallel"; that one is easy to understand.
• The second is "tests within a single test collection do not run in parallel against each other, but tests from different test collections can run in parallel against each other". Further, we have a flag together on test collections which says "this test collection cannot be run in parallel against other test collections", so in any given project, there are 0 or more test collections opting out of parallelism entirely.

Additionally, we the two issues linked again are requests for:

• Run everything in parallel, always. (I assume there would need to be some opt out here as well, like we have with test collections. The mechanism here is TBD.)
• If tests don't have any shared context (no fixtures), they're eligible for being run in parallel against any other test, even in the same test class; if there are fixtures, then the old rules apply.

Now that we're at (at least) four different ways of defining how to run things in parallel, it feels like the old way of specifying parallelism isn't sufficient. So I think I want to change how we specify intra-assembly parallelism in configuration (remove parallelizeTestCollections) and command line switches (rework -parallel), and instead replace it with a user-customizable "parallelism sorter".

I'm not 100% sure whether this design is where I'll land, but it's a starting point for discussion.

The component would implement some interface, and be registered at the assembly-level. It would me given a list of all the test cases, and it would sort them into groups which serve two functions: (a) zero or one "non-parallel" group, which means any test case in that group is not run in parallel against any other test (this is to accommodate the "non-parallel" test collections today, as well as being extended to perhaps the test class and/or test method level); (b) zero or more "parallel" groups which contain tests which cannot be run in parallel against each other within the same group, but can be run in parallel against any test in any other "parallel" group.

For the purposes of illustration below, let's assume that there are ways to opt out of parallelism on a per-test-collection basis (that exists today, via CollectionDefinition.DisableParallelization = true) as well as perhaps on a per-test-class and per-test-method basis. We'll just call those "non-parallel tests" for simplicity.

The default behaviors would work like this:

Nothing runs in parallel

Everything is placed into the "non-parallel" group. No "parallel" groups exist.

Test collections run in parallel against each other

The "non-parallel tests" go into the "non-parallel" group. Create one "parallel" group per remaining test collection, containing all the tests in that collection.

The new behaviors could be accommodated with:

Everything runs in parallel

The "non-parallel tests" go into the "non-parallel" group. Create one "parallel" group per remaining test.

Disable parallelism only when there is shared context

The "non-parallel tests" go into the "non-parallel" group. For every test that has some shared fixture instance (excepting assembly-level fixtures), group them together into a "parallel" group, per shared fixture instance. For every test that does not have any fixtures, create one "parallel" group per test.

I think the "shared fixture instance" is a bit complicated, so here's my thinking about this.

We share instances of fixture data from ICollectionFixture<> and IClassFixture<>. That means in this definition, anybody in a test collection with any instances of ICollectionFixture<> attached to it end up in a group together (because they're sharing the collection fixture instance). Then, any test classes which have IClassFixture<> (either directly or via a collection definition) but no ICollectionFixture<> would cause all the tests in that class to be in a group together.

It's important to remember that this is about shared fixture instances. That means, in the scenario below, we'd end up with two "parallel" groups here, not one, because despite sharing a fixture type, the don't share a fixture instance:

public class TestClass1 : IClassFixture<MyFixture> { }

public class TestClass2 : IClassFixture<MyFixture> { }

Ditto for multiple test collections which both use the same fixture type via ICollectionFixture<>, since they'll be sharing the type but not the instance.

That's the sum total of my thoughts. Removing this as a configuration item with built-in behavior, and instead replacing it with a user-customizable, compile-time (assembly-level) choice, should allow us to be able to add new rule sets later without mucking around with configuration options. It should also allow users to do unusual things that seem right for them, like my (currently over complicated) sample of parallelizing based on namespace, which would become fairly trivial in the new design.

So here are my open questions:

1. What do you think of this design?
   a. Is the design reasonable, or too complex? Do you have an alternate design that's cleaner/simpler while still allowing for all the requirements?
   b. Do you have parallelization requirements that are different than the four options shown here? Can you accomplish those goals with the generalized design here?

2. What do you think of the idea of removing the intra-assembly parallelism options (configuration files, command line switches, etc.)? What about keeping the inter-assembly parallelism options (for the multi-assembly runners like our first party Console and MSBuild runners)? Should we also remove those configuration file options and leave this decision solely to the runner based on command line options?