[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
------------------------------------
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection<DynamicNode> createDynamicTests() {
List<DynamicNode> folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List<DynamicNode> folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List<DynamicNode> rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest (pkg)
** createDynamicTests()
*** folder1
**** folder2
***** test1
***** test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates content in surefire-reports like so
{code}
<testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
time="0.012"/>
<testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
time="0.001"/>
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
<testcase name="folder2 folder1 createDynamicTests() test1"
classname="pkg.JUnit5DynamicTest" time="0.011"/>
<testcase name="folder2 folder1 createDynamicTests() test2"
classname="pkg.JUnit5DynamicTest" time="0.001"/>
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the phrased "method" name. Also the internal ordering seems
inconsistent, if we expect the ordering defined by heirarchy position then we
would expect the folder/method names like "createDynamicTest() folder1 folder2
test1".
Problem is how do the tools that process this XML into HTML reports handle it?
* should we enable phrased naming to expose the
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
<testcase name="test1"
classname="pkg.SampleTest.createDynamicTests().aFolder.Tests"/>
<testcase name="test2"
classname="pkg.SampleTest.createDynamicTests().aFolder.Tests"/>
{code}
Or if we considered the java package is a distraction when using dynamic tests
{code:java}
<testcase name="test1" classname="aFolder.Tests"/>
<testcase name="test2" classname="aFolder.Tests"/>
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
...
public class SampleTest {
@TestFactory
Collection<DynamicContainer> createDynamicTests() {
List<DynamicTest> testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List<DynamicContainer> root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
**** test1
**** test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
<testcase name="aFolder createDynamicTests() test1"
classname="pkg.SampleTest" time="0.02"/>
<testcase name="aFolder createDynamicTests() test2"
classname="pkg.SampleTest" time="0.001"/>
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
<testcase name="test1"
classname="pkg.SampleTest.createDynamicTests().aFolder.Tests"/>
<testcase name="test2"
classname="pkg.SampleTest.createDynamicTests().aFolder.Tests"/>
{code}
Or if we considered the java package is a distraction when using dynamic tests
{code:java}
<testcase name="test1" classname="aFolder.Tests"/>
<testcase name="test2" classname="aFolder.Tests"/>
{code}
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---------------------------------------------------------------------------
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M7
> Reporter: Geoff Soutter
> Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection<DynamicNode> createDynamicTests() {
> List<DynamicNode> folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List<DynamicNode> folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List<DynamicNode> rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest (pkg)
> ** createDynamicTests()
> *** folder1
> **** folder2
> ***** test1
> ***** test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates content in surefire-reports like
> so
> {code}
> <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
> time="0.012"/>
> <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
> time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
> <testcase name="folder2 folder1 createDynamicTests() test1"
> classname="pkg.JUnit5DynamicTest" time="0.011"/>
> <testcase name="folder2 folder1 createDynamicTests() test2"
> classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report. The DynamicContainer and DynamicTest names are
> injected into the phrased "method" name. Also the internal ordering seems
> inconsistent, if we expect the ordering defined by heirarchy position then we
> would expect the folder/method names like "createDynamicTest() folder1
> folder2 test1".
> Problem is how do the tools that process this XML into HTML reports handle it?
> * should we enable phrased naming to expose the
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
> <testcase name="test1"
> classname="pkg.SampleTest.createDynamicTests().aFolder.Tests"/>
> <testcase name="test2"
> classname="pkg.SampleTest.createDynamicTests().aFolder.Tests"/>
> {code}
> Or if we considered the java package is a distraction when using dynamic tests
> {code:java}
> <testcase name="test1" classname="aFolder.Tests"/>
> <testcase name="test2" classname="aFolder.Tests"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
