[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". However, things are not as good when using surefire as it still uses the years old ant XSD for XML reporting and that XSD doesn't support anything apart from basic Java usage, certainly nothing like dynamic tests which don't correspond directly to a java class and method name, Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates XML report content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it (eg Jenkins JUnit plugin). * should surefire uses be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * how can XML reporting tools show the nested structure of the DynamicFolders? ** can XML reporting tools easily tell whether the XML was generated with phrased or non phrased settings? Should they need to? ** if using phrased, should they parse the phrased name attribute to find the dynamic folder and test names? ** if using non phrased, is there a way of surefire safely injecting the dynamic folder and test names into an existing attribute which will work in a backwards compatible way with the XML reporting tools? Also the internal ordering of the phrased name attribute seems inconsistent, or at least I don't understand it. Since "test" is inside "folder1", surely "folder1" and "test" should be next to each other rather than at opposite ends of the value? If we expect ordering by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas Perhaps one day https://github.com/ota4j-team/open-test-reporting will be supported as XML report format and at that point, I guess support for dynamic tests would be easy to add. However that could be years off, if at all. Anyway, back to the real world, Surefire using the old ant XSD can generate two variants of the XML report, using non phrased or phrased names. Being able to control the phrased and non phrased attributes individually seems to make no sense as the attributes are related. But anyway, we have to make do. Seems for JUnit5 XML reporting, the intention for the "name" attribute value was: * non phrased is the TestIdentifier.legacyReportingName * phrased is the TestIdentifier.displayName h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. The JUnit5 legacyReportingName for the example looks like "createDynamicTests[1][1][1]". However this is basically no use as it uses synthetic ids rather than the real names and will end up as a flat list without any hierarchical structure in XML reporting tools. From memory. this was actually used in earlier surefire
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". However, things are not as good when using surefire as it still uses the years old ant XSD for XML reporting and that XSD doesn't support anything apart from basic Java usage, certainly nothing like dynamic tests which don't correspond directly to a java class and method name, Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates XML report content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it (eg Jenkins JUnit plugin). * should surefire uses be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * how can XML reporting tools show the nested structure of the DynamicFolders? ** can XML reporting tools easily tell whether the XML was generated with phrased or non phrased settings? Should they need to? ** if using phrased, should they parse the phrased name attribute to find the dynamic folder and test names? ** if using non phrased, is there a way of surefire safely injecting the dynamic folder and test names into an existing attribute which will work in a backwards compatible way with the XML reporting tools? Also the internal ordering of the phrased name attribute seems inconsistent, or at least I don't understand it. Since "test" is inside "folder1", surely "folder1" and "test" should be next to each other rather than at opposite ends of the value? If we expect ordering by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas Perhaps one day https://github.com/ota4j-team/open-test-reporting will be supported as XML report format and at that point, I guess support for dynamic tests would be easy to add. However that could be years off, if at all. Anyway, back to the real world, Surefire using the old ant XSD has two variants of the XML report, using non phrased method names, or phrased method names. Dynamic tests don't have a non phrased name. So its not really clear to me what the distinction between phrased and non phrased names means for Dynamic tests. Nevertheless, phrased and non phrased is what Surefire is designed for, so we have to come up with something. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. JUnit5 provides a legacyReportingName e.g. "createDynamicTests[1][1][1]" which I think was intended to use in the name attribute of the old XML report format. However this is basically no use as it uses synthetic ids rather than the real names and will end up as a flat list without any hierarchical structure in XML reporting tools. I think this was actually used in earlier surefire
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". However, things are not as good when using surefire as it still uses the years old ant XSD for XML reporting and that XSD doesn't support anything apart from basic Java usage, certainly nothing like dynamic tests which don't correspond directly to a java class and method name, Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates XML report content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it (eg Jenkins JUnit plugin). * should surefire uses be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * how can XML reporting tools show the nested structure of the DynamicFolders? ** can XML reporting tools easily tell whether the XML was generated with phrased or non phrased settings? Should they need to? ** if using phrased, should they parse the phrased name attribute to find the dynamic folder and test names? ** if using non phrased, is there a way of surefire safely injecting the dynamic folder and test names into an existing attribute which will work in a backwards compatible way with the XML reporting tools? Also the internal ordering of the phrased name attribute seems inconsistent, or at least I don't understand it. Since "test" is inside "folder1", surely "folder1" and "test" should be next to each other rather than at opposite ends of the value? If we expect ordering by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas Perhaps one day https://github.com/ota4j-team/open-test-reporting will be supported as XML report format and at that point, I guess support for dynamic tests would be easy to add. However that could be years off, if at all. Anyway, back to the real world, Surefire using the old ant XSD has two variants of the XML report, using non phrased method names, or phrased method names. Dynamic tests don't have a non phrased name. So its not really clear to me what the distinction between phrased and non phrased names means for Dynamic tests. Nevertheless, phrased and non phrased is what Surefire is designed for, so we have to come up with something. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". However, things are not as good when using surefire as it still uses the years old ant XSD for XML reporting and that XSD doesn't support anything apart from basic Java usage, certainly nothing like dynamic tests which don't correspond directly to a java class and method name, Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates XML report content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it (eg Jenkins JUnit plugin). * should surefire uses be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * how can XML reporting tools show the nested structure of the DynamicFolders? ** can XML reporting tools easily tell whether the XML was generated with phrased or non phrased settings? Should they need to? ** if using phrased, should they parse the phrased name attribute to find the dynamic folder and test names? ** if using non phrased, is there a way of surefire safely injecting the dynamic folder and test names into an existing attribute which will work in a backwards compatible way with the XML reporting tools? Also the internal ordering of the phrased name attribute seems inconsistent, or at least I don't understand the ordering. Surely folder1 should come before folder2 If we expect ordering by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas Perhaps one day https://github.com/ota4j-team/open-test-reporting will be supported as XML report format and at that point, I guess support for dynamic tests would be easy to add. However that could be years off, if at all. Anyway, back to the real world, Surefire using the old ant XSD has two variants of the XML report, using non phrased method names, or phrased method names. Dynamic tests don't have a non phrased name. So its not really clear to me what the distinction between phrased and non phrased names means for Dynamic tests. Nevertheless, phrased and non phrased is what Surefire is designed for, so we have to come up with something. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". However, things are not as good when using surefire as it still uses the years old ant XSD for XML reporting and that XSD doesn't support anything apart from basic Java usage, certainly nothing like dynamic tests which don't correspond directly to a java class and method name, Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates XML report content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it? * should we be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * assuming the above is true, how can reporting tools show the nested structure of the DynamicFolders? Should they parse the phrased method name to find the individual components? Also the internal ordering seems inconsistent, or at least I don't understand the structure. surely folder1 should come before folder2 If we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas Perhaps one day https://github.com/ota4j-team/open-test-reporting will be supported as XML report format and at that point, I guess support for dynamic tests would be easy to add. However that could be years off, if at all. Anyway, back to the real world, Surefire using the old ant XSD has two variants of the XML report, using non phrased method names, or phrased method names. Dynamic tests don't have a non phrased name. So its not really clear to me what the distinction between phrased and non phrased names means for Dynamic tests. Nevertheless, phrased and non phrased is what Surefire is designed for, so we have to come up with something. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} h3. Phrased For phrased usage, its less obvious what to do. The nested structure is injected into a single XML attribute and I haven't seen any documentation of the internal structure of that attribute. The ordering looks inconsistent as it is and possibly needs fixing. However, changing this value without any schema as to
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". However, things are not as good when using surefire as it still uses the years old ant XSD for XML reporting and that XSD doesn't support anything apart from basic Java usage, certainly nothing like dynamic tests which don't correspond directly to a java class and method name, Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates XML report content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it? * should we be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * assuming the above is true, how can reporting tools show the nested structure of the DynamicFolders? Should they parse the phrased method name to find the individual components? Also the internal ordering seems inconsistent, or at least I don't understand the structure. surely folder1 should come before folder2 If we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas We have two ways to look at this, using non phrased method names, or phrased method names. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} h3. Phrased For phrased usage, its less obvious what to do. The nested structure is injected into a single XML attribute and I haven't seen any documentation of the internal structure of that attribute. The ordering looks inconsistent as it is and possibly needs fixing. However, changing this value without any schema as to what the contents mean seems dangerous. Casual testing with an available Jenkins instance seems to indicate that Jenkins JUnit already has some capability to parse the phrased method name as it shows folder2 as a "package" (but folder1 is lost). To get an overall useful result here I guess we'd need to * create and publish some kind of docs / schema on the internal structure we want to support for downstream reporting tools * adjust whatever output we have now to meet the docs/schema, if required * wait for downstream reporting tools to support it
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Priority: Major (was: Minor) > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7, 3.2.5 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package pkg; > import ... > public class JUnit5DynamicTest { > @TestFactory > Collection createDynamicTests() { > List folder2 = new ArrayList<>(); > folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List folder1 = new ArrayList<>(); > folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); > List rootFolder = new ArrayList<>(); > rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); > return rootFolder; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * JUnit5DynamicTest (pkg) > ** createDynamicTests() > *** folder1 > folder2 > * test1 > * test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" > reported configuration disabled, it creates content in surefire-reports like > so > {code} >time="0.012"/> >time="0.001"/> > {code} > In this case the Dynamic Tests are completely obscured, instead we get > duplicate reports for the @TestFactory method. > With the example "phrased" reporter configuration "use" properties enabled, > we get > {code:java} >classname="pkg.JUnit5DynamicTest" time="0.011"/> >classname="pkg.JUnit5DynamicTest" time="0.001"/> > {code} > Here all the Dynamic tests are created as synthetic "methods" of the physical > class in the HTML report, with the DynamicContainer and DynamicTest names > injected into the phrased "method" name. > From the perspective of someone with a lot of dynamic tests in a huge folder > structure, this is less than ideal as the nested structure is not exposed in > the test report in a clean structure way, as it was with package names. > So how do the tools that process this XML into HTML reports handle it? > * should we be forced to enable phrased naming so that reporting tools can > access the dynamic folder and test names? > * assuming the above is true, how can reporting tools show the nested > structure of the DynamicFolders? Should they parse the phrased method name to > find the individual components? > Also the internal ordering seems inconsistent, or at least I don't understand > the structure. surely folder1 should come before folder2 If we expect the > ordering defined by heirarchy position then we would expect the folder/method > names like "createDynamicTest() folder1 folder2 test1". > h3. Solution Ideas > We have two ways to look at this, using non phrased method names, or phrased > method names. > h3. Non Phrased > Dynamic tests with non phrased method names are currently not usable. So > anything we do here would be an improvement. > So for non phrased method names, we could add the dynamic names as synthetic > child packages in the classname attribute. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | class name | static placeholder (e.g. Tests) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/> >classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/> > {code} > h3. Phrased > For phrased usage, its less obvious what to do. > The nested structure is injected into a single XML attribute and I haven't > seen any documentation of the internal structure of that attribute. > The ordering looks inconsistent as it is and possibly needs fixing. However, > changing
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Priority: Minor (was: Major) > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7, 3.2.5 >Reporter: Geoff Soutter >Priority: Minor > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package pkg; > import ... > public class JUnit5DynamicTest { > @TestFactory > Collection createDynamicTests() { > List folder2 = new ArrayList<>(); > folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List folder1 = new ArrayList<>(); > folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); > List rootFolder = new ArrayList<>(); > rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); > return rootFolder; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * JUnit5DynamicTest (pkg) > ** createDynamicTests() > *** folder1 > folder2 > * test1 > * test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" > reported configuration disabled, it creates content in surefire-reports like > so > {code} >time="0.012"/> >time="0.001"/> > {code} > In this case the Dynamic Tests are completely obscured, instead we get > duplicate reports for the @TestFactory method. > With the example "phrased" reporter configuration "use" properties enabled, > we get > {code:java} >classname="pkg.JUnit5DynamicTest" time="0.011"/> >classname="pkg.JUnit5DynamicTest" time="0.001"/> > {code} > Here all the Dynamic tests are created as synthetic "methods" of the physical > class in the HTML report, with the DynamicContainer and DynamicTest names > injected into the phrased "method" name. > From the perspective of someone with a lot of dynamic tests in a huge folder > structure, this is less than ideal as the nested structure is not exposed in > the test report in a clean structure way, as it was with package names. > So how do the tools that process this XML into HTML reports handle it? > * should we be forced to enable phrased naming so that reporting tools can > access the dynamic folder and test names? > * assuming the above is true, how can reporting tools show the nested > structure of the DynamicFolders? Should they parse the phrased method name to > find the individual components? > Also the internal ordering seems inconsistent, or at least I don't understand > the structure. surely folder1 should come before folder2 If we expect the > ordering defined by heirarchy position then we would expect the folder/method > names like "createDynamicTest() folder1 folder2 test1". > h3. Solution Ideas > We have two ways to look at this, using non phrased method names, or phrased > method names. > h3. Non Phrased > Dynamic tests with non phrased method names are currently not usable. So > anything we do here would be an improvement. > So for non phrased method names, we could add the dynamic names as synthetic > child packages in the classname attribute. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | class name | static placeholder (e.g. Tests) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/> >classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/> > {code} > h3. Phrased > For phrased usage, its less obvious what to do. > The nested structure is injected into a single XML attribute and I haven't > seen any documentation of the internal structure of that attribute. > The ordering looks inconsistent as it is and possibly needs fixing. However, > changing
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Affects Version/s: 3.2.5 > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7, 3.2.5 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package pkg; > import ... > public class JUnit5DynamicTest { > @TestFactory > Collection createDynamicTests() { > List folder2 = new ArrayList<>(); > folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List folder1 = new ArrayList<>(); > folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); > List rootFolder = new ArrayList<>(); > rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); > return rootFolder; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * JUnit5DynamicTest (pkg) > ** createDynamicTests() > *** folder1 > folder2 > * test1 > * test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" > reported configuration disabled, it creates content in surefire-reports like > so > {code} >time="0.012"/> >time="0.001"/> > {code} > In this case the Dynamic Tests are completely obscured, instead we get > duplicate reports for the @TestFactory method. > With the example "phrased" reporter configuration "use" properties enabled, > we get > {code:java} >classname="pkg.JUnit5DynamicTest" time="0.011"/> >classname="pkg.JUnit5DynamicTest" time="0.001"/> > {code} > Here all the Dynamic tests are created as synthetic "methods" of the physical > class in the HTML report, with the DynamicContainer and DynamicTest names > injected into the phrased "method" name. > From the perspective of someone with a lot of dynamic tests in a huge folder > structure, this is less than ideal as the nested structure is not exposed in > the test report in a clean structure way, as it was with package names. > So how do the tools that process this XML into HTML reports handle it? > * should we be forced to enable phrased naming so that reporting tools can > access the dynamic folder and test names? > * assuming the above is true, how can reporting tools show the nested > structure of the DynamicFolders? Should they parse the phrased method name to > find the individual components? > Also the internal ordering seems inconsistent, or at least I don't understand > the structure. surely folder1 should come before folder2 If we expect the > ordering defined by heirarchy position then we would expect the folder/method > names like "createDynamicTest() folder1 folder2 test1". > h3. Solution Ideas > We have two ways to look at this, using non phrased method names, or phrased > method names. > h3. Non Phrased > Dynamic tests with non phrased method names are currently not usable. So > anything we do here would be an improvement. > So for non phrased method names, we could add the dynamic names as synthetic > child packages in the classname attribute. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | class name | static placeholder (e.g. Tests) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/> >classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/> > {code} > h3. Phrased > For phrased usage, its less obvious what to do. > The nested structure is injected into a single XML attribute and I haven't > seen any documentation of the internal structure of that attribute. > The ordering looks inconsistent as it is and possibly needs fixing. However, > changing this
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it? * should we be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * assuming the above is true, how can reporting tools show the nested structure of the DynamicFolders? Should they parse the phrased method name to find the individual components? Also the internal ordering seems inconsistent, or at least I don't understand the structure. surely folder1 should come before folder2 If we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas We have two ways to look at this, using non phrased method names, or phrased method names. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} h3. Phrased For phrased usage, its less obvious what to do. The nested structure is injected into a single XML attribute and I haven't seen any documentation of the internal structure of that attribute. The ordering looks inconsistent as it is and possibly needs fixing. However, changing this value without any schema as to what the contents mean seems dangerous. Casual testing with an available Jenkins instance seems to indicate that Jenkins JUnit already has some capability to parse the phrased method name as it shows folder2 as a "package" (but folder1 is lost). To get an overall useful result here I guess we'd need to * create and publish some kind of docs / schema on the internal structure we want to support for downstream reporting tools * adjust whatever output we have now to meet the docs/schema, if required * wait for downstream reporting tools to support it was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>();
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it? * should we be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * assuming the above is true, how can reporting tools show the nested structure of the DynamicFolders? Should they parse the phrased method name to find the individual components? Also the internal ordering seems inconsistent, if we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas We have two ways to look at this, using non phrased method names, or phrased method names. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} h3. Phrased For phrased usage, its less obvious what to do. The nested structure is injected into a single XML attribute and I haven't seen any documentation of the internal structure of that attribute. The ordering looks inconsistent as it is and possibly needs fixing. However, changing this value without any schema as to what the contents mean seems to be dangerous. Casual testing with an available Jenkins instance seems to indicate that Jenkins JUnit already has some capability to parse the phrased method name as it shows folder2 as a "package" (but folder1 is lost). To get an overall useful result here I guess we'd need to * create and publish some kind of docs / schema on the internal structure we want to support for downstream reporting tools * adjust whatever output we have now to meet the docs/schema, if required * wait for downstream reporting tools to support it was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it? * should we be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * assuming the above is true, how can reporting tools show the nested structure of the DynamicFolders? Should they parse the phrased method name to find the individual components? Also the internal ordering seems inconsistent, if we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas We have two ways to look at this, using non phrased method names, or phrased method names. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} h3. Phrased For phrased usage, its less obvious what to do. The nested structure is injected into a single XML attribute and I haven't seen any documentation of the internal structure of that attribute. The ordering looks inconsistent as it is and possibly needs fixing. However, changing this value without any schema as to what the contents mean seems to be dangerous. Casual testing with an available Jenkins instance seems to indicate that Jenkins JUnit already has some capability to parse the phrased method name as it shows folder2 as a "package" (but folder1 is lost). Getting an overall useful result here would mostly rely on fixes in the downstream reporting tools. was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>();
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * JUnit5DynamicTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report, with the DynamicContainer and DynamicTest names injected into the phrased "method" name. >From the perspective of someone with a lot of dynamic tests in a huge folder >structure, this is less than ideal as the nested structure is not exposed in >the test report in a clean structure way, as it was with package names. So how do the tools that process this XML into HTML reports handle it? * should we be forced to enable phrased naming so that reporting tools can access the dynamic folder and test names? * assuming the above is true, how can reporting tools show the nested structure of the DynamicFolders? Should they parse the phrased method name to find the individual components? Also the internal ordering seems inconsistent, if we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". h3. Solution Ideas We have two ways to look at this, using non phrased method names, or phrased method names. h3. Non Phrased Dynamic tests with non phrased method names are currently not usable. So anything we do here would be an improvement. So for non phrased method names, we could add the dynamic names as synthetic child packages in the classname attribute. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} h3. Phrased For phrased usage, its less obvious what to do. The nested structure is injected into a single XML attribute and I haven't seen any documentation of the internal structure of that attribute. The ordering looks inconsistent as it is and possibly needs fixing. However, changing this value without any schema as to what the contents mean seems to be dangerous. Casual testing with an available Jenkins instance seems to indicate that Jenkins JUnit already has some capability to parse the phrased method name as it shows folder2 as a "package" (but folder1 is lost). Getting an overall useful result here would mostly rely on fixes in the downstream reporting tools. was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; import ... public class JUnit5DynamicTest { @TestFactory Collection createDynamicTests() { List folder2 = new ArrayList<>(); folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable())); folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List folder1 = new ArrayList<>(); folder1.add(DynamicContainer.dynamicContainer("folder2", folder2)); List rootFolder = new ArrayList<>(); rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1)); return rootFolder; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest (pkg) ** createDynamicTests() *** folder1 folder2 * test1 * test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" reported configuration disabled, it creates content in surefire-reports like so {code} {code} In this case the Dynamic Tests are completely obscured, instead we get duplicate reports for the @TestFactory method. With the example "phrased" reporter configuration "use" properties enabled, we get {code:java} {code} Here all the Dynamic tests are created as synthetic "methods" of the physical class in the HTML report. The DynamicContainer and DynamicTest names are injected into the phrased "method" name. Also the internal ordering seems inconsistent, if we expect the ordering defined by heirarchy position then we would expect the folder/method names like "createDynamicTest() folder1 folder2 test1". Problem is how do the tools that process this XML into HTML reports handle it? * should we enable phrased naming to expose the This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} Or if we considered the java package is a distraction when using dynamic tests {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package pkg; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} Or if we considered the java package is a distraction when using dynamic tests {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} Or if we considered the java package is a distraction when using dynamic tests {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Tests) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} Or if we considered the java package irrelevant in this situation {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Affects Version/s: 3.0.0-M7 > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 3.0.0-M7 >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package samplepackage; > ... > public class SampleTest { > @TestFactory > Collection createDynamicTests() { > List testList = new ArrayList<>(); > testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List root = new ArrayList<>(); > root.add(DynamicContainer.dynamicContainer("aFolder", testList)); > return root; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following hierarchical structure in the UI > treeview > * SampleTest > ** createDynamicTests() > *** aFolder > test1 > test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.0.0-m7, with the example "phrased" reporter configuration > enabled, it creates contents in surefire-reports like so: > {code:java} >classname="samplepackage.SampleTest" time="0.02"/> >classname="samplepackage.SampleTest" time="0.001"/> > {code} > Here all the nested synthetic tests are created as synthetic "methods" of a > single class in the HTML report. The DynamicContainer and DynamicTest names > are injected into the synthetic "method" name. That is, from a reporting > perspective, they are created as a flat list (also it seems the folder name > is in the wrong place, it seems it should be on the left of the method name, > eg "createDynamicTests() aFolder test1" in order for the report to match the > internal structure). > This doesn't scale nicely as the number of tests in the dynamically created > "folder" structure grows. We can't view the different DynamicContainers > separately in the rendered HTML reports. We can't get a breakdown on any > particular DynamicContainer in the rendered HTML reports. > h3. Solution Ideas > I think what we want here is that dynamic tests using dynamic containers > should have those container names structured as synthetic packages. This > allows us the benefits of having the nested structure modelled correctly in > the HTML report. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | simple class name | static placeholder (e.g. Test) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test" > time="0.02"/> >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test" > time="0.001"/> > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as synthetic "methods" of a single class in the HTML report. The DynamicContainer and DynamicTest names are injected into the synthetic "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Test) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as the child of a single class in the HTML report. The folder name is injected into the "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as the child of a single class in the HTML report. The folder name is injected into the "method" name. That is, from a reporting perspective, they are created as a flat list (also it seems the folder name is in the wrong place, it seems it should be on the left of the method name, eg "createDynamicTests() aFolder test1" in order for the report to match the internal structure). This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Test) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as the child of a single class in the HTML report. The folder name is injected into the "method" name. That is, from a reporting perspective, they are created as a flat list. This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Description: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following hierarchical structure in the UI treeview * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as the child of a single class in the HTML report. The folder name is injected into the "method" name. That is, from a reporting perspective, they are created as a flat list. This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Test) | | method name | DynamicTest name | I think we could do that by moving the method name and folder name from name= into classname= and adding a static placeholder for the classname. {code:java} {code} was: h3. Description of the issue I created a simple example of JUnit5 DynamicContainer/DynamicTest {code:java} package samplepackage; ... public class SampleTest { @TestFactory Collection createDynamicTests() { List testList = new ArrayList<>(); testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); List root = new ArrayList<>(); root.add(DynamicContainer.dynamicContainer("aFolder", testList)); return root; } private static class MyExecutable implements Executable { @Override public void execute() throws Throwable { } } } {code} When I run this in IDEA. I get the following structure * SampleTest ** createDynamicTests() *** aFolder test1 test2 Here we can see that dynamic tests are represented by IDEA as nested children of the test method. We can open and close the "virtual packages" created as usual to explore in the dynamically created test results. It is not a "flat list". Using surefire 3.0.0-m7, with the example "phrased" reporter configuration enabled, it creates contents in surefire-reports like so: {code:java} {code} Here all the nested synthetic tests are created as the child of a single class in the HTML report. The folder name is injected into the "method" name. That is, from a reporting perspective, they are created as a flat list. This doesn't scale nicely as the number of tests in the dynamically created "folder" structure grows. We can't view the different DynamicContainers separately in the rendered HTML reports. We can't get a breakdown on any particular DynamicContainer in the rendered HTML reports. h3. Solution Ideas I think what we want here is that dynamic tests using dynamic containers should have those container names structured as synthetic packages. This allows us the benefits of having the nested structure modelled correctly in the HTML report. That is, when mapping from DynamicTests to Java packages, we have: || Java Structure || Dynamic Test Structure || | package identifier | DynamicContainer name | | simple class name | static placeholder (e.g. Test) | | method name | DynamicTest name | I think we could
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoff Soutter updated SUREFIRE-2147: Summary: JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure (was: Report for JUnit5 DynamicContainer/DynamicTest loses the hierarchical structure) > JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure > --- > > Key: SUREFIRE-2147 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2147 > Project: Maven Surefire > Issue Type: Bug >Reporter: Geoff Soutter >Priority: Major > > h3. Description of the issue > I created a simple example of JUnit5 DynamicContainer/DynamicTest > {code:java} > package samplepackage; > ... > public class SampleTest { > @TestFactory > Collection createDynamicTests() { > List testList = new ArrayList<>(); > testList.add(DynamicTest.dynamicTest("test1", new MyExecutable())); > testList.add(DynamicTest.dynamicTest("test2", new MyExecutable())); > List root = new ArrayList<>(); > root.add(DynamicContainer.dynamicContainer("aFolder", testList)); > return root; > } > private static class MyExecutable implements Executable { > @Override > public void execute() throws Throwable { > } > } > } > {code} > When I run this in IDEA. I get the following structure > * SampleTest > ** createDynamicTests() > *** aFolder > test1 > test2 > Here we can see that dynamic tests are represented by IDEA as nested children > of the test method. We can open and close the "virtual packages" created as > usual to explore in the dynamically created test results. It is not a "flat > list". > Using surefire 3.0.0-m7, with the example "phrased" reporter configuration > enabled, it creates contents in surefire-reports like so: > {code:java} >classname="samplepackage.SampleTest" time="0.02"/> >classname="samplepackage.SampleTest" time="0.001"/> > {code} > Here all the nested synthetic tests are created as the child of a single > class in the HTML report. The folder name is injected into the "method" name. > That is, from a reporting perspective, they are created as a flat list. > This doesn't scale nicely as the number of tests in the dynamically created > "folder" structure grows. We can't view the different DynamicContainers > separately in the rendered HTML reports. We can't get a breakdown on any > particular DynamicContainer in the rendered HTML reports. > h3. Solution Ideas > I think what we want here is that dynamic tests using dynamic containers > should have those container names structured as synthetic packages. This > allows us the benefits of having the nested structure modelled correctly in > the HTML report. > That is, when mapping from DynamicTests to Java packages, we have: > || Java Structure || Dynamic Test Structure || > | package identifier | DynamicContainer name | > | simple class name | static placeholder (e.g. Test) | > | method name | DynamicTest name | > > I think we could do that by moving the method name and folder name from name= > into classname= and adding a static placeholder for the classname. > {code:java} >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test" > time="0.02"/> >classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test" > time="0.001"/> > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)