[ 
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoff Soutter updated SUREFIRE-2147:
------------------------------------
    Description: 
h3. Description of the issue

I created a simple example of JUnit5 DynamicContainer/DynamicTest

{code:java}
package pkg;

import ...

public class JUnit5DynamicTest {

    @TestFactory
    Collection<DynamicNode> createDynamicTests() {

        List<DynamicNode> folder2 = new ArrayList<>();
        folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
        folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));

        List<DynamicNode> folder1 = new ArrayList<>();
        folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));

        List<DynamicNode> rootFolder = new ArrayList<>();
        rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
        return rootFolder;

    }

    private static class MyExecutable implements Executable {
        @Override
        public void execute() throws Throwable {
        }
    }

}
{code}

When I run this in IDEA. I get the following hierarchical structure in the UI 
treeview

 * JUnit5DynamicTest (pkg)
 ** createDynamicTests()
 *** folder1
 **** folder2
 ***** test1
 ***** test2

Here we can see that dynamic tests are represented by IDEA as nested children 
of the test method. We can open and close the "virtual packages" created as 
usual to explore in the dynamically created test results. It is not a "flat 
list".

However, things are not as good when using surefire as it still uses the years 
old ant XSD for XML reporting and that XSD doesn't support anything apart from 
basic Java usage, certainly nothing like dynamic tests which don't correspond 
directly to a java class and method name,

Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" 
reported configuration disabled, it creates XML report content in 
surefire-reports like so

{code}
  <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
time="0.012"/>
  <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
time="0.001"/>
{code}

In this case the Dynamic Tests are completely obscured, instead we get 
duplicate reports for the @TestFactory method. 

With the example "phrased" reporter configuration "use" properties enabled, we 
get
{code:java}
  <testcase name="folder2 folder1 createDynamicTests() test1" 
classname="pkg.JUnit5DynamicTest" time="0.011"/>
  <testcase name="folder2 folder1 createDynamicTests() test2" 
classname="pkg.JUnit5DynamicTest" time="0.001"/>
{code}

Here all the Dynamic tests are created as synthetic "methods" of the physical 
class in the HTML report, with the DynamicContainer and DynamicTest names 
injected into the phrased "method" name. 

>From the perspective of someone with a lot of dynamic tests in a huge folder 
>structure, this is less than ideal as the nested structure is not exposed in 
>the test report in a clean structure way, as it was with package names.

So how do the tools that process this XML into HTML reports handle it? 

* should we be forced to enable phrased naming so that reporting tools can 
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure 
of the DynamicFolders? Should they parse the phrased method name to find the 
individual components?

Also the internal ordering seems inconsistent, or at least I don't understand 
the structure. surely folder1 should come before folder2 If we expect the 
ordering defined by heirarchy position then we would expect the folder/method 
names like "createDynamicTest() folder1 folder2 test1".

h3. Solution Ideas

We have two ways to look at this, using non phrased method names, or phrased 
method names.

h3. Non Phrased

Dynamic tests with non phrased method names are currently not usable. So 
anything we do here would be an improvement.

So for non phrased method names, we could add the dynamic names as synthetic 
child packages in the classname attribute. 

That is, when mapping from DynamicTests to Java packages, we have:

|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
  
I think we could do that by moving the method name and folder name from name= 
into classname= and adding a static placeholder for the classname.
{code:java}
  <testcase name="test1" 
classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
  <testcase name="test2" 
classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
{code}

h3. Phrased

For phrased usage, its less obvious what to do. 

The nested structure is injected into a single XML attribute and I haven't seen 
any documentation of the internal structure of that attribute. 

The ordering looks inconsistent as it is and possibly needs fixing. However, 
changing this value without any schema as to what the contents mean seems 
dangerous. 

Casual testing with an available Jenkins instance seems to indicate that 
Jenkins JUnit already has some capability to parse the phrased method name as 
it shows folder2 as a "package" (but folder1 is lost). 

To get an overall useful result here I guess we'd need to

* create and publish some kind of docs / schema on the internal structure we 
want to support for downstream reporting tools
* adjust whatever output we have now to meet the docs/schema, if required
* wait for downstream reporting tools to support it

  was:
h3. Description of the issue

I created a simple example of JUnit5 DynamicContainer/DynamicTest

{code:java}
package pkg;

import ...

public class JUnit5DynamicTest {

    @TestFactory
    Collection<DynamicNode> createDynamicTests() {

        List<DynamicNode> folder2 = new ArrayList<>();
        folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
        folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));

        List<DynamicNode> folder1 = new ArrayList<>();
        folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));

        List<DynamicNode> rootFolder = new ArrayList<>();
        rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
        return rootFolder;

    }

    private static class MyExecutable implements Executable {
        @Override
        public void execute() throws Throwable {
        }
    }

}
{code}

When I run this in IDEA. I get the following hierarchical structure in the UI 
treeview

 * JUnit5DynamicTest (pkg)
 ** createDynamicTests()
 *** folder1
 **** folder2
 ***** test1
 ***** test2

Here we can see that dynamic tests are represented by IDEA as nested children 
of the test method. We can open and close the "virtual packages" created as 
usual to explore in the dynamically created test results. It is not a "flat 
list".

Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" 
reported configuration disabled, it creates content in surefire-reports like so

{code}
  <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
time="0.012"/>
  <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
time="0.001"/>
{code}

In this case the Dynamic Tests are completely obscured, instead we get 
duplicate reports for the @TestFactory method. 

With the example "phrased" reporter configuration "use" properties enabled, we 
get
{code:java}
  <testcase name="folder2 folder1 createDynamicTests() test1" 
classname="pkg.JUnit5DynamicTest" time="0.011"/>
  <testcase name="folder2 folder1 createDynamicTests() test2" 
classname="pkg.JUnit5DynamicTest" time="0.001"/>
{code}

Here all the Dynamic tests are created as synthetic "methods" of the physical 
class in the HTML report, with the DynamicContainer and DynamicTest names 
injected into the phrased "method" name. 

>From the perspective of someone with a lot of dynamic tests in a huge folder 
>structure, this is less than ideal as the nested structure is not exposed in 
>the test report in a clean structure way, as it was with package names.

So how do the tools that process this XML into HTML reports handle it? 

* should we be forced to enable phrased naming so that reporting tools can 
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure 
of the DynamicFolders? Should they parse the phrased method name to find the 
individual components?

Also the internal ordering seems inconsistent, or at least I don't understand 
the structure. surely folder1 should come before folder2 If we expect the 
ordering defined by heirarchy position then we would expect the folder/method 
names like "createDynamicTest() folder1 folder2 test1".

h3. Solution Ideas

We have two ways to look at this, using non phrased method names, or phrased 
method names.

h3. Non Phrased

Dynamic tests with non phrased method names are currently not usable. So 
anything we do here would be an improvement.

So for non phrased method names, we could add the dynamic names as synthetic 
child packages in the classname attribute. 

That is, when mapping from DynamicTests to Java packages, we have:

|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
  
I think we could do that by moving the method name and folder name from name= 
into classname= and adding a static placeholder for the classname.
{code:java}
  <testcase name="test1" 
classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
  <testcase name="test2" 
classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
{code}

h3. Phrased

For phrased usage, its less obvious what to do. 

The nested structure is injected into a single XML attribute and I haven't seen 
any documentation of the internal structure of that attribute. 

The ordering looks inconsistent as it is and possibly needs fixing. However, 
changing this value without any schema as to what the contents mean seems 
dangerous. 

Casual testing with an available Jenkins instance seems to indicate that 
Jenkins JUnit already has some capability to parse the phrased method name as 
it shows folder2 as a "package" (but folder1 is lost). 

To get an overall useful result here I guess we'd need to

* create and publish some kind of docs / schema on the internal structure we 
want to support for downstream reporting tools
* adjust whatever output we have now to meet the docs/schema, if required
* wait for downstream reporting tools to support it


> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---------------------------------------------------------------------------
>
>                 Key: SUREFIRE-2147
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 3.0.0-M7, 3.2.5
>            Reporter: Geoff Soutter
>            Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
>     @TestFactory
>     Collection<DynamicNode> createDynamicTests() {
>         List<DynamicNode> folder2 = new ArrayList<>();
>         folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
>         folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
>         List<DynamicNode> folder1 = new ArrayList<>();
>         folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
>         List<DynamicNode> rootFolder = new ArrayList<>();
>         rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
>         return rootFolder;
>     }
>     private static class MyExecutable implements Executable {
>         @Override
>         public void execute() throws Throwable {
>         }
>     }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI 
> treeview
>  * JUnit5DynamicTest (pkg)
>  ** createDynamicTests()
>  *** folder1
>  **** folder2
>  ***** test1
>  ***** test2
> Here we can see that dynamic tests are represented by IDEA as nested children 
> of the test method. We can open and close the "virtual packages" created as 
> usual to explore in the dynamically created test results. It is not a "flat 
> list".
> However, things are not as good when using surefire as it still uses the 
> years old ant XSD for XML reporting and that XSD doesn't support anything 
> apart from basic Java usage, certainly nothing like dynamic tests which don't 
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" 
> reported configuration disabled, it creates XML report content in 
> surefire-reports like so
> {code}
>   <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
> time="0.012"/>
>   <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
> time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get 
> duplicate reports for the @TestFactory method. 
> With the example "phrased" reporter configuration "use" properties enabled, 
> we get
> {code:java}
>   <testcase name="folder2 folder1 createDynamicTests() test1" 
> classname="pkg.JUnit5DynamicTest" time="0.011"/>
>   <testcase name="folder2 folder1 createDynamicTests() test2" 
> classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical 
> class in the HTML report, with the DynamicContainer and DynamicTest names 
> injected into the phrased "method" name. 
> From the perspective of someone with a lot of dynamic tests in a huge folder 
> structure, this is less than ideal as the nested structure is not exposed in 
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it? 
> * should we be forced to enable phrased naming so that reporting tools can 
> access the dynamic folder and test names?
> * assuming the above is true, how can reporting tools show the nested 
> structure of the DynamicFolders? Should they parse the phrased method name to 
> find the individual components?
> Also the internal ordering seems inconsistent, or at least I don't understand 
> the structure. surely folder1 should come before folder2 If we expect the 
> ordering defined by heirarchy position then we would expect the folder/method 
> names like "createDynamicTest() folder1 folder2 test1".
> h3. Solution Ideas
> We have two ways to look at this, using non phrased method names, or phrased 
> method names.
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So 
> anything we do here would be an improvement.
> So for non phrased method names, we could add the dynamic names as synthetic 
> child packages in the classname attribute. 
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>   
> I think we could do that by moving the method name and folder name from name= 
> into classname= and adding a static placeholder for the classname.
> {code:java}
>   <testcase name="test1" 
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
>   <testcase name="test2" 
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> h3. Phrased
> For phrased usage, its less obvious what to do. 
> The nested structure is injected into a single XML attribute and I haven't 
> seen any documentation of the internal structure of that attribute. 
> The ordering looks inconsistent as it is and possibly needs fixing. However, 
> changing this value without any schema as to what the contents mean seems 
> dangerous. 
> Casual testing with an available Jenkins instance seems to indicate that 
> Jenkins JUnit already has some capability to parse the phrased method name as 
> it shows folder2 as a "package" (but folder1 is lost). 
> To get an overall useful result here I guess we'd need to
> * create and publish some kind of docs / schema on the internal structure we 
> want to support for downstream reporting tools
> * adjust whatever output we have now to meet the docs/schema, if required
> * wait for downstream reporting tools to support it



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to