[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815514#comment-17815514 ] Geoff Soutter commented on SUREFIRE-2147: - This would help with making DynamicTests more usable in IDEs. https://github.com/junit-team/junit5/issues/3682 > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7, 3.2.5 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package pkg; > import ... > public class JUnit5DynamicTest { > @TestFactory > Collection createDynamicTests() { > List folder2 = new ArrayList<>(); > folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List folder1 = new ArrayList<>(); > folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); > List rootFolder = new ArrayList<>(); > rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); > return rootFolder; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * JUnit5DynamicTest (pkg) > ** createDynamicTests() > *** folder1 > folder2 > * test1 > * test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > However, things are not as good when using surefire as it still uses the > years old ant XSD for XML reporting and that XSD doesn't support anything > apart from basic Java usage, certainly nothing like dynamic tests which don't > correspond directly to a java class and method name, > Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" > reported configuration disabled, it creates XML report content in > surefire-reports like so > {code} >time="0.012"/> >time="0.001"/> > {code} > In this case the Dynamic Tests are completely obscured, instead we get > duplicate reports for the @TestFactory method. > With the example "phrased" reporter configuration "use" properties enabled, > we get > {code:java} >classname="pkg.JUnit5DynamicTest" time="0.011"/> >classname="pkg.JUnit5DynamicTest" time="0.001"/> > {code} > Here all the Dynamic tests are created as synthetic "methods" of the physical > class in the HTML report, with the DynamicContainer and DynamicTest names > injected into the phrased "method" name. > From the perspective of someone with a lot of dynamic tests in a huge folder > structure, this is less than ideal as the nested structure is not exposed in > the test report in a clean structure way, as it was with package names. > So how do the tools that process this XML into HTML reports handle it (eg > Jenkins JUnit plugin). > * should surefire uses be forced to enable phrased naming so that reporting > tools can access the dynamic folder and test names? > * how can XML reporting tools show the nested structure of the > DynamicFolders? > ** can XML reporting tools easily tell whether the XML was generated with > phrased or non phrased settings? Should they need to? > ** if using phrased, should they parse the phrased name attribute to find the > dynamic folder and test names? > ** if using non phrased, is there a way of surefire safely injecting the > dynamic folder and test names into an existing attribute which will work in a > backwards compatible way with the XML reporting tools? > Also the internal ordering of the phrased name attribute seems inconsistent, > or at least I don't understand it. Since "test" is inside "folder1", surely > "folder1" and "test" should be next to each other rather than at opposite > ends of the value? If we expect ordering by heirarchy position then we would > expect the folder/method names like "createDynamicTest() folder1 folder2 > test1". > h3. Solution Ideas > Perhaps one day https://github.com/ota4j-team/open-test-reporting will be > supported as XML report format and at that point, I guess support for dynamic > tests would be easy to add. However that could be years off, if at all. > Anyway, back to the real world, Surefire using the old ant XSD can generate > two variants of the XML report,
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815095#comment-17815095 ] Geoff Soutter commented on SUREFIRE-2147: - [~tibordigana] [~marcphilipp] Are you able to provide any design guidance on this? > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7, 3.2.5 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package pkg; > import ... > public class JUnit5DynamicTest { > @TestFactory > Collection createDynamicTests() { > List folder2 = new ArrayList<>(); > folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List folder1 = new ArrayList<>(); > folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); > List rootFolder = new ArrayList<>(); > rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); > return rootFolder; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * JUnit5DynamicTest (pkg) > ** createDynamicTests() > *** folder1 > folder2 > * test1 > * test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > However, things are not as good when using surefire as it still uses the > years old ant XSD for XML reporting and that XSD doesn't support anything > apart from basic Java usage, certainly nothing like dynamic tests which don't > correspond directly to a java class and method name, > Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" > reported configuration disabled, it creates XML report content in > surefire-reports like so > {code} >time="0.012"/> >time="0.001"/> > {code} > In this case the Dynamic Tests are completely obscured, instead we get > duplicate reports for the @TestFactory method. > With the example "phrased" reporter configuration "use" properties enabled, > we get > {code:java} >classname="pkg.JUnit5DynamicTest" time="0.011"/> >classname="pkg.JUnit5DynamicTest" time="0.001"/> > {code} > Here all the Dynamic tests are created as synthetic "methods" of the physical > class in the HTML report, with the DynamicContainer and DynamicTest names > injected into the phrased "method" name. > From the perspective of someone with a lot of dynamic tests in a huge folder > structure, this is less than ideal as the nested structure is not exposed in > the test report in a clean structure way, as it was with package names. > So how do the tools that process this XML into HTML reports handle it (eg > Jenkins JUnit plugin). > * should surefire uses be forced to enable phrased naming so that reporting > tools can access the dynamic folder and test names? > * how can XML reporting tools show the nested structure of the > DynamicFolders? > ** can XML reporting tools easily tell whether the XML was generated with > phrased or non phrased settings? Should they need to? > ** if using phrased, should they parse the phrased name attribute to find the > dynamic folder and test names? > ** if using non phrased, is there a way of surefire safely injecting the > dynamic folder and test names into an existing attribute which will work in a > backwards compatible way with the XML reporting tools? > Also the internal ordering of the phrased name attribute seems inconsistent, > or at least I don't understand the ordering. Surely folder1 should come > before folder2 If we expect ordering by heirarchy position then we would > expect the folder/method names like "createDynamicTest() folder1 folder2 > test1". > h3. Solution Ideas > Perhaps one day https://github.com/ota4j-team/open-test-reporting will be > supported as XML report format and at that point, I guess support for dynamic > tests would be easy to add. However that could be years off, if at all. > Anyway, back to the real world, Surefire using the old ant XSD has two > variants of the XML report, using non phrased method names, or phrased method > names. > Dynamic tests don't have a non phrased name. So its not really
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808437#comment-17808437 ] Geoff Soutter commented on SUREFIRE-2147: - Had a quick look at 3.2.5 to see if any changes. Not sure if I noticed before, but tried 3.2.5 on a real use case, which maps a folder structure into Dynamic tests. Therefore I have parent Containers nested inside child Containers. The current implementation seems to throw away the parent containers, only the leaf container is shown in the report. I think I can live with the rest of the behaviour, but throwing away "parent" container names fundamentally breaks my use case. Hopefully I can set aside some time to do contribute a PR for this (although I've promised that a few times already and yet to deliver anything. Sigh). > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package samplepackage; > ... > public class SampleTest { > @TestFactory > Collection createDynamicTests() { > List testList = new ArrayList<>(); > testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List root = new ArrayList<>(); > root.add(DynamicContainer.dynamicContainer("aFolder", testList)); > return root; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * SampleTest > ** createDynamicTests() > *** aFolder > test1 > test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.0.0-m7, with the example "phrased" reporter configuration > enabled, it creates contents in surefire-reports like so: > {code:java} >classname="samplepackage.SampleTest" time="0.02"/> >classname="samplepackage.SampleTest" time="0.001"/> > {code} > Here all the nested synthetic tests are created as synthetic "methods" of a > single class in the HTML report. The DynamicContainer and DynamicTest names > are injected into the synthetic "method" name. That is, from a reporting > perspective, they are created as a flat list (also it seems the folder name > is in the wrong place, it seems it should be on the left of the method name, > eg "createDynamicTests() aFolder test1" in order for the report to match the > internal structure). > This doesn't scale nicely as the number of tests in the dynamically created > "folder" structure grows. We can't view the different DynamicContainers > separately in the rendered HTML reports. We can't get a breakdown on any > particular DynamicContainer in the rendered HTML reports. > h3. Solution Ideas > I think what we want here is that dynamic tests using dynamic containers > should have those container names structured as synthetic packages. This > allows us the benefits of having the nested structure modelled correctly in > the HTML report. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | simple class name | static placeholder (e.g. Tests) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/> >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/> > {code} > Or if we considered the java package is a distraction when using dynamic tests > {code:java} > > > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579 ] Geoff Soutter commented on SUREFIRE-2147: - Checked surefire 3.2.2 and seems this has been mostly fixed. Whereas before we get this {code} {code} Looks like now we get {code} {code} This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML and create a hierarchical report. In case of test being in the root folder, it is inconsistent. But I suspect few would have that scenario, {code} {code} Not quite sure what fixed it, but I suspect changes for SUREFIRE-2117, which was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks [~sjaranowski]. > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package samplepackage; > ... > public class SampleTest { > @TestFactory > Collection createDynamicTests() { > List testList = new ArrayList<>(); > testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List root = new ArrayList<>(); > root.add(DynamicContainer.dynamicContainer("aFolder", testList)); > return root; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * SampleTest > ** createDynamicTests() > *** aFolder > test1 > test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.0.0-m7, with the example "phrased" reporter configuration > enabled, it creates contents in surefire-reports like so: > {code:java} >classname="samplepackage.SampleTest" time="0.02"/> >classname="samplepackage.SampleTest" time="0.001"/> > {code} > Here all the nested synthetic tests are created as synthetic "methods" of a > single class in the HTML report. The DynamicContainer and DynamicTest names > are injected into the synthetic "method" name. That is, from a reporting > perspective, they are created as a flat list (also it seems the folder name > is in the wrong place, it seems it should be on the left of the method name, > eg "createDynamicTests() aFolder test1" in order for the report to match the > internal structure). > This doesn't scale nicely as the number of tests in the dynamically created > "folder" structure grows. We can't view the different DynamicContainers > separately in the rendered HTML reports. We can't get a breakdown on any > particular DynamicContainer in the rendered HTML reports. > h3. Solution Ideas > I think what we want here is that dynamic tests using dynamic containers > should have those container names structured as synthetic packages. This > allows us the benefits of having the nested structure modelled correctly in > the HTML report. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | simple class name | static placeholder (e.g. Test) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test" > time="0.02"/> >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test" > time="0.001"/> > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)