[JUnit Platform Engine] Add Surefire naming strategy (#3003)

When testing with Surefire using the `long` naming strategy the feature
name is included in every error message twice. This can be quite
annoying when the name is long. For example:

```
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.103 s <<< FAILURE! -- in io.cucumber.skeleton.RunCucumberTest
[ERROR] Belly.Belly - a few cukes -- Time elapsed: 0.060 s <<< FAILURE!
java.lang.AssertionError:
        at io.cucumber.skeleton.StepDefinitions.I_have_cukes_in_my_belly(StepDefinitions.java:11)
        at ✽.I have 42 cukes in my belly(classpath:io/cucumber/skeleton/belly.feature:4)
```

Like wise the xml report by Surefire includes the name twice.

```xml
<testcase name="Belly - a few cukes" classname="Belly" time="0.06">
    <failure type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError:
	at io.cucumber.skeleton.StepDefinitions.I_have_cukes_in_my_belly(StepDefinitions.java:11)
	at ✽.I have 42 cukes in my belly(classpath:io/cucumber/skeleton/belly.feature:4)
]]></failure>
```

By using the `cucumber.junit-platform.naming-strategy=surefire` property
when configuring surefire tests will look pretty in surefire too.

```xml
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.5.2</version>
    <configuration>
        <properties>
            <!-- Work around. Surefire does not include enough
                 information to disambiguate between different
                 examples and scenarios. -->
            <configurationParameters>
                cucumber.junit-platform.naming-strategy=surefire
            </configurationParameters>
        </properties>
    </configuration>
</plugin>
```

For example:

```
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.102 s <<< FAILURE! -- in io.cucumber.skeleton.RunCucumberTest
[ERROR] Belly.a few cukes -- Time elapsed: 0.059 s <<< FAILURE!
java.lang.AssertionError: TODO
        at io.cucumber.skeleton.StepDefinitions.I_have_cukes_in_my_belly(StepDefinitions.java:
```

And

```xml
<testcase name="a few cukes" classname="Belly" time="0.059">
    <failure message="TODO" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: TODO
	at io.cucumber.skeleton.StepDefinitions.I_have_cukes_in_my_belly(StepDefinitions.java:10)
	at ✽.I have 42 cukes in my belly(classpath:io/cucumber/skeleton/belly.feature:4)
]]></failure>
```

Author: mpkorstanje

CHANGELOG.md

@@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 ### Added
 - [JUnit Platform Engine, TestNG] Remove framework elements from `UndefinedStepException` stacktrace ([#3002](https://github.com/cucumber/cucumber-jvm/pull/3002) M.P. Korstanje)
+- [JUnit Platform Engine] Add `surefire` naming strategy ([#3003](https://github.com/cucumber/cucumber-jvm/pull/3003) M.P. Korstanje)
+
+### Changed
+- [JUnit Platform Engine] Use `number-and-pickle-if-parameterized ` example naming strategy by default ([#3004](https://github.com/cucumber/cucumber-jvm/pull/3004) M.P. Korstanje)
 
 ## [7.22.2] - 2025-05-12
 ### Changed

cucumber-archetype/src/main/resources/archetype-resources/pom.xml

@@ -83,7 +83,7 @@
                              information to disambiguate between different
                              examples and scenarios. -->
                         <configurationParameters>
-                            cucumber.junit-platform.naming-strategy=long
+                            cucumber.junit-platform.naming-strategy=surefire
                         </configurationParameters>
                     </properties>
                 </configuration>

cucumber-archetype/src/test/resources/projects/should-generate-project/reference/pom.xml

@@ -83,7 +83,7 @@
                              information to disambiguate between different
                              examples and scenarios. -->
                         <configurationParameters>
-                            cucumber.junit-platform.naming-strategy=long
+                            cucumber.junit-platform.naming-strategy=surefire
                         </configurationParameters>
                     </properties>
                 </configuration>

cucumber-junit-platform-engine/README.md

@@ -54,9 +54,11 @@ for a brief how to.
 
 Because Surefire and Gradle reports provide the results in a `<Class Name> - <Method Name>`
 format, only scenario names or example numbers are reported. This
-can make for hard to read reports. To improve the readability of the reports
-provide the `cucumber.junit-platform.naming-strategy=long` configuration
-parameter. This will include the feature name as part of the test name. 
+can make for hard to read reports. 
+
+To improve the readability of the reports provide the
+`cucumber.junit-platform.naming-strategy` configuration parameter to
+Gradle. This will include the feature name as part of the test name.
 
 ##### Maven
 
@@ -68,7 +70,7 @@ parameter. This will include the feature name as part of the test name.
     <configuration>
         <properties>
             <configurationParameters>
-                cucumber.junit-platform.naming-strategy=long
+                cucumber.junit-platform.naming-strategy=surefire
             </configurationParameters>
         </properties>
     </configuration>
@@ -416,7 +418,7 @@ cucumber.filter.tags=                                          # a cucumber tag
 cucumber.glue=                                                 # comma separated package names.
                                                                # example: com.example.glue  
 
-cucumber.junit-platform.naming-strategy=                       # long or short.
+cucumber.junit-platform.naming-strategy=                       # long, short or surefire.
                                                                # default: short
                                                                # include parent descriptor name in test descriptor.
 
@@ -429,6 +431,11 @@ cucumber.junit-platform.naming-strategy.long.example-name=     # number, number-
                                                                # default: number-and-pickle-if-parameterized
                                                                # Use example number or pickle name for examples when
                                                                # long naming strategy is used
+                                                               
+cucumber.junit-platform.naming-strategy.surefire.example-name= # number or pickle.
+                                                               # default: number-and-pickle-if-parameterized
+                                                               # Use example number or pickle name for examples when
+                                                               # surefire naming strategy is used
 
 cucumber.plugin=                                               # comma separated plugin strings.
                                                                # example: pretty, json:path/to/report.json
@@ -615,7 +622,7 @@ Note: any files written by Cucumber will be overwritten during the rerun.
                  information to disambiguate between different
                  examples and scenarios. -->
             <configurationParameters>
-                cucumber.junit-platform.naming-strategy=long
+                cucumber.junit-platform.naming-strategy=surefire
             </configurationParameters>
         </properties>
     </configuration>

cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/Constants.java

@@ -126,15 +126,20 @@ public final class Constants {
     /**
      * Property name used to configure the naming strategy: {@value}
      * <p>
-     * Value must be one of {@code long} or {@code short}. By default, short
-     * names are used.
+     * Value must be one of {@code long}, {@code short}, or {@code surefire}. By
+     * default, short names are used.
      * <p>
-     * When long names are used the parent descriptor names are included into
-     * each test descriptor name. So for example a single example would be
-     * named:
+     * When the {@code long} naming strategy is used all parent descriptor names
+     * are included in each test descriptor name. So for example a single
+     * example would be named:
      * {@code Feature Name - Rule Name - Scenario Name - Examples Name - Example #N }.
-     * This is useful for tools that only report the test name such as Maven and
-     * Gradle.
+     * This is useful for tools that only report the test name such as Gradle.
+     * <p>
+     * When the {@code surefire} naming strategy is used nodes are named such
+     * that the Surefire output makes sense. The feature name will be rendered
+     * as the class name. The long name without the feature will be rendered as
+     * the test method name. For example:
+     * {@code Feature Name.Rule Name - Scenario Name - Examples Name - Example #N}.
      */
     @API(status = Status.EXPERIMENTAL, since = "7.0.0")
     public static final String JUNIT_PLATFORM_NAMING_STRATEGY_PROPERTY_NAME = "cucumber.junit-platform.naming-strategy";
@@ -160,6 +165,27 @@ public final class Constants {
     @API(status = Status.EXPERIMENTAL, since = "7.16.2")
     public static final String JUNIT_PLATFORM_SHORT_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME = "cucumber.junit-platform.naming-strategy.short.example-name";
 
+    /**
+     * Property name used to configure the naming strategy of examples in case
+     * of surefire naming strategy: {@value}
+     * <p>
+     * Value must be one of {@code number}, {@code pickle}, or
+     * {@code number-and-pickle-if-parameterized}. By default,
+     * {@code number-and-pickle-if-parameterized} is used.
+     * <ul>
+     * <li>When set to {@code number} examples are numbered. So the first
+     * example of the first examples section would be named {@code #1.1}
+     * <li>When set to {@code pickle} the pickle name is used. So for scenario
+     * name {@code Adding <a> and <b>} and example with params {@code a = 10}
+     * and {@code b = 20} the following name would be produced:
+     * {@code Adding 10 and 20}.
+     * <li>When set to {@code number-and-pickle-if-parameterized} the name would
+     * be rendered as {@code #1.1: Adding 10 and 20}.
+     * </ul>
+     */
+    @API(status = Status.EXPERIMENTAL, since = "7.23.0")
+    public static final String JUNIT_PLATFORM_SUREFIRE_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME = "cucumber.junit-platform.naming-strategy.surefire.example-name";
+
     /**
      * Property name used to configure the naming strategy of examples in case
      * of long naming strategy: {@value}

cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/DefaultNamingStrategyProvider.java

@@ -11,6 +11,7 @@
 
 import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_LONG_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_SHORT_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME;
+import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_SUREFIRE_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME;
 
 enum DefaultNamingStrategyProvider {
     LONG {
@@ -31,6 +32,16 @@ NamingStrategy create(ConfigurationParameters configuration) {
                     .orElse(DefaultNamingStrategyProvider::exampleNumberAndPickleIfParameterizedStrategy)
                     .apply(DefaultNamingStrategyProvider::shortStrategy);
         }
+    },
+
+    SUREFIRE {
+        @Override
+        NamingStrategy create(ConfigurationParameters configuration) {
+            return configuration.get(JUNIT_PLATFORM_SUREFIRE_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME)
+                    .map(DefaultNamingStrategyProvider::parseStrategy)
+                    .orElse(DefaultNamingStrategyProvider::exampleNumberAndPickleIfParameterizedStrategy)
+                    .apply(DefaultNamingStrategyProvider::surefireStrategy);
+        }
     };
 
     abstract NamingStrategy create(ConfigurationParameters configuration);
@@ -125,4 +136,43 @@ private static String longStrategy(Node node, String currentNodeName) {
 
         return builder.toString();
     }
+
+    private static String surefireStrategy(Node node, String currentNodeName) {
+        // Surefire uses the parents of test nodes to determine the class name.
+        // As we want the class name to match the feature name we name the
+        // parents of the test containing nodes after the feature.
+        if (node instanceof Node.Examples || node instanceof Node.Rule) {
+            return nameOrKeyword(findFeature(node));
+        }
+        // Scenarios and examples names are used by surefire to populate the
+        // testname. We want this to be long, but without the feature name
+        // because that will be used for the class name
+        if (node instanceof Node.Scenario || node instanceof Node.Example) {
+            return longStrategyWithoutFeatureName(node, currentNodeName);
+        }
+        // Everything else, can be short, will be ignored by surefire.
+        return shortStrategy(node, currentNodeName);
+    }
+
+    private static String longStrategyWithoutFeatureName(Node node, String currentNodeName) {
+        StringBuilder builder = new StringBuilder();
+        builder.append(currentNodeName);
+        node = node.getParent().orElse(null);
+
+        while (node != null && !(node instanceof Node.Feature)) {
+            builder.insert(0, " - ");
+            builder.insert(0, nameOrKeyword(node));
+            node = node.getParent().orElse(null);
+        }
+        return builder.toString();
+    }
+
+    private static Node findFeature(Node node) {
+        Node candidate = node.getParent().orElse(null);
+        while (candidate != null) {
+            node = candidate;
+            candidate = node.getParent().orElse(null);
+        }
+        return node;
+    }
 }

cucumber-junit-platform-engine/src/test/java/io/cucumber/junit/platform/engine/FeatureResolverTest.java

@@ -220,6 +220,31 @@ void shortNamesWithPickleNamesIfParameterized() {
         assertEquals("Example #1.1: A scenario with A", example.getDisplayName());
     }
 
+    @Test
+    void surefireNamesWithPickleNamesIfParameterized() {
+        configurationParameters = new MapConfigurationParameters(
+            JUNIT_PLATFORM_NAMING_STRATEGY_PROPERTY_NAME, "surefire");
+
+        // The test node gets the long name, without the feature name.
+        TestDescriptor example = getParametrizedExample();
+        assertEquals("A scenario with <example> - Examples - Example #1.1: A scenario with A",
+            example.getDisplayName());
+
+        // The parent node gets the feature name.
+        TestDescriptor examples = example.getParent().get();
+        assertEquals("A feature with scenario outlines",
+            examples.getDisplayName());
+
+        // Remaining nodes are named by their short names.
+        TestDescriptor scenarioOutline = examples.getParent().get();
+        assertEquals("A scenario with <example>",
+            scenarioOutline.getDisplayName());
+
+        TestDescriptor feature = scenarioOutline.getParent().get();
+        assertEquals("A feature with scenario outlines",
+            feature.getDisplayName());
+    }
+
     private TestDescriptor getExample() {
         return getOutline().getChildren().iterator().next().getChildren().iterator().next();
     }

cucumber-junit-platform-engine/src/test/resources/io/cucumber/junit/platform/engine/feature-with-outline.feature

@@ -48,3 +48,15 @@ Feature: A feature with scenario outlines
       | example |
       | A       |
       | B       |
+  
+  @ScenarioOutlineTag
+  Scenario Outline: A scenario with <example>
+    Given a parameterized scenario outline
+    When it is executed
+    Then <example> is used
+
+    @Example1Tag
+    Examples:
+      | example |
+      | A       |
+      | B       |

examples/calculator-java-junit5/pom.xml

@@ -92,7 +92,7 @@
                         <!-- Work around. Surefire does not include enough information to disambiguate between different
                              examples. -->
                         <configurationParameters>
-                            cucumber.junit-platform.naming-strategy=long
+                            cucumber.junit-platform.naming-strategy=surefire
                         </configurationParameters>
                     </properties>
                 </configuration>

examples/calculator-kotlin-junit5/pom.xml

@@ -114,7 +114,7 @@
                         <!-- Work around. Surefire does not include enough information to disambiguate between different
                              examples. -->
                         <configurationParameters>
-                            cucumber.junit-platform.naming-strategy=long
+                            cucumber.junit-platform.naming-strategy=surefire
                         </configurationParameters>
                     </properties>
                 </configuration>

examples/spring-java-junit5/pom.xml

@@ -111,7 +111,7 @@
                         <!-- Work around. Surefire does not include enough information to disambiguate between different
                              examples. -->
                         <configurationParameters>
-                            cucumber.junit-platform.naming-strategy=long
+                            cucumber.junit-platform.naming-strategy=surefire
                         </configurationParameters>
                     </properties>
                 </configuration>