Support for cargo-nextest and other test runners

[eglif-roche]

Cucumber only works with cargo test, but not with other test runners (e.g. cargo-nextest).

What would be required to support this:

Nextest calls the test executable first with the parameters --list --format terse. This should print a list of all tests.
e.g.

scenario_name1: test
scenario_name2: test
scenario_name3: test

Then Nextest calls the test executed with the parameters --list --ignored --format terse. This should print a list of all currently ignored tests.

Then Nextest calls the test executable with the parameter --exact <scenario_name> . This should execute only the specified test.

As a workaround we had to add a wrapper and CLI around Cucumber to support this. But it would be nice if this would be supported out of the box by Cucumber.

[tyranron]

The relevant docs: https://github.com/nextest-rs/nextest/blob/cargo-nextest-0.9.94/site/src/docs/design/custom-test-harnesses.md#manually-implementing-a-test-harness

[tyranron]

@eglif-roche well, from what you've described and nextest documentation suggests, the way it works, basically:

1. Runs the test binary with --list --format terse args and receives the list of tests:

   scenario_name1: test
   scenario_name2: test
   scenario_name3: test
   

2. Then for each test in this list executes <test-name> --nocapture --exact.

This "process-per-scenario" approach does not apply well to Cucumber at all. There are at least several questions about this:

• To run <test-name> --nocapture --exact we basically need to run through all the parsed scenarios but execute only the one filtered by its name. And do this every time for each scenario. For example, our codebase having ~2250 scenarios at the moment would require to redo parsing 2250 times. Which benefit of nextest outweighs it?
• --ignored semantics are unclear. In Cucumber we have "filtered out" (the ones not executed at all) and "skipped" (the ones executed, but not fully defined) scenarios. I'm uncertain what to mean here as "ignored". Again, to observe "skipped" ones, we need kinda "execute" them (at least run through all its steps to detect whether they have matchers defined).
• Implied semantics breakage. "process-per-scenario" approach will break any meaning of things like @serial, they just won't work, and we won't be able to make them work since we have no control over how nextest runs test executables inside this executable. Also, any baked-in summarization will break too, since each scenario is run as a separate executable now.

Regarding these question, I'm not so inclined to make out-of-the-box support for nextest, because people will be able to use it easily and then complain on broken semantics/promises, performance problems. If someone really needs it, they will better do it via custom CLI as you've done to support their concrete case.
Supporting this properly in common case seems possible, but via different very dirty hacks (like test executables communicating with each other via external files using flocks for serializing execution) and substantial amount of effort. Again, which benefit of nextest for Cucumber outweighs it?

@ilslv @bbqsrc I would be happy to hear your opinion on this too.

[eglif-roche]

Due to the missing support for JUnit test result files in "cargo test" we rely on other test runners in our CI/CD pipelines.
Although Cucumber supports JUnit XML reports and would work for our cucumber tests, this does not cover the other unit and integration tests.

But your argument about the performance is valid.

[tyranron]

@eglif-roche well, you could separate Cucumber tests into their own workspace crate and run them via cargo test, while anything other in the workspace run as cargo nextest. That's not something unusual, and we tend to do this in our codebases all the time (1, 2), because this way it's much less interfering between dev-dependencies and dependencies of the main crate (usually, the dependencies for the app/lib are other than for E2E-testing them, so why bother recompiling everything every time?).