[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:31 AM:
-------------------------------------------------------------------
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
<testcase name="aFolder createDynamicTests() test1"
classname="samplepackage.SampleTest"/>
<testcase name="aFolder createDynamicTests() test2"
classname="samplepackage.SampleTest"/>
{code}
Looks like now we get
{code}
<testcase name="test1" classname="aFolder"/>
<testcase name="test2" classname="aFolder"/>
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
<testcase name="test1" classname="createDynamicTests()"/>
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
<testcase name="aFolder createDynamicTests() test1"
classname="samplepackage.SampleTest"/>
<testcase name="aFolder createDynamicTests() test2"
classname="samplepackage.SampleTest"/>
{code}
Looks like now we get
{code}
<testcase name="test1" classname="aFolder" time="0.02"/>
<testcase name="test2" classname="aFolder " time="0.001"/>
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
<testcase name="test1" classname="createDynamicTests()" time="0.02"/>
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---------------------------------------------------------------------------
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M7
> Reporter: Geoff Soutter
> Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection<DynamicContainer> createDynamicTests() {
> List<DynamicTest> testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List<DynamicContainer> root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> **** test1
> **** test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
> <testcase name="aFolder createDynamicTests() test1"
> classname="samplepackage.SampleTest" time="0.02"/>
> <testcase name="aFolder createDynamicTests() test2"
> classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
> <testcase name="test1"
> classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
> <testcase name="test2"
> classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
> {code}
> Or if we considered the java package is a distraction when using dynamic tests
> {code:java}
> <testcase name="test1" classname="aFolder.Tests"/>
> <testcase name="test2" classname="aFolder.Tests"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
