[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815514#comment-17815514
]
Geoff Soutter commented on SUREFIRE-2147:
-
This would help with making DynamicTests more usable in IDEs.
https://github.com/junit-team/junit5/issues/3682
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> However, things are not as good when using surefire as it still uses the
> years old ant XSD for XML reporting and that XSD doesn't support anything
> apart from basic Java usage, certainly nothing like dynamic tests which don't
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates XML report content in
> surefire-reports like so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the
> DynamicFolders?
> ** can XML reporting tools easily tell whether the XML was generated with
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the
> dynamic folder and test names into an existing attribute which will work in a
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent,
> or at least I don't understand it. Since "test" is inside "folder1", surely
> "folder1" and "test" should be next to each other rather than at opposite
> ends of the value? If we expect ordering by heirarchy position then we would
> expect the folder/method names like "createDynamicTest() folder1 folder2
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
> supported as XML report format and at that point, I guess support for dynamic
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD can generate
> two variants of the XML report,
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815095#comment-17815095
]
Geoff Soutter commented on SUREFIRE-2147:
-
[~tibordigana] [~marcphilipp] Are you able to provide any design guidance on
this?
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> However, things are not as good when using surefire as it still uses the
> years old ant XSD for XML reporting and that XSD doesn't support anything
> apart from basic Java usage, certainly nothing like dynamic tests which don't
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates XML report content in
> surefire-reports like so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the
> DynamicFolders?
> ** can XML reporting tools easily tell whether the XML was generated with
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the
> dynamic folder and test names into an existing attribute which will work in a
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent,
> or at least I don't understand the ordering. Surely folder1 should come
> before folder2 If we expect ordering by heirarchy position then we would
> expect the folder/method names like "createDynamicTest() folder1 folder2
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
> supported as XML report format and at that point, I guess support for dynamic
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD has two
> variants of the XML report, using non phrased method names, or phrased method
> names.
> Dynamic tests don't have a non phrased name. So its not really
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808437#comment-17808437
]
Geoff Soutter commented on SUREFIRE-2147:
-
Had a quick look at 3.2.5 to see if any changes.
Not sure if I noticed before, but tried 3.2.5 on a real use case, which maps a
folder structure into Dynamic tests. Therefore I have parent Containers nested
inside child Containers. The current implementation seems to throw away the
parent containers, only the leaf container is shown in the report.
I think I can live with the rest of the behaviour, but throwing away "parent"
container names fundamentally breaks my use case.
Hopefully I can set aside some time to do contribute a PR for this (although
I've promised that a few times already and yet to deliver anything. Sigh).
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
> {code}
> Or if we considered the java package is a distraction when using dynamic tests
> {code:java}
>
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter commented on SUREFIRE-2147:
-
Checked surefire 3.2.2 and seems this has been mostly fixed.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
In case of test being in the root folder, it is inconsistent. But I suspect few
would have that scenario,
{code}
{code}
Not quite sure what fixed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
