[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815514#comment-17815514
]
Geoff Soutter commented on SUREFIRE-2147:
-
This would help with making DynamicTests more usable in IDEs.
https://github.com/junit-team/junit5/issues/3682
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> However, things are not as good when using surefire as it still uses the
> years old ant XSD for XML reporting and that XSD doesn't support anything
> apart from basic Java usage, certainly nothing like dynamic tests which don't
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates XML report content in
> surefire-reports like so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the
> DynamicFolders?
> ** can XML reporting tools easily tell whether the XML was generated with
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the
> dynamic folder and test names into an existing attribute which will work in a
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent,
> or at least I don't understand it. Since "test" is inside "folder1", surely
> "folder1" and "test" should be next to each other rather than at opposite
> ends of the value? If we expect ordering by heirarchy position then we would
> expect the folder/method names like "createDynamicTest() folder1 folder2
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
> supported as XML report format and at that point, I guess support for dynamic
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD can generate
> two variants of the XML report,
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
However, things are not as good when using surefire as it still uses the years
old ant XSD for XML reporting and that XSD doesn't support anything apart from
basic Java usage, certainly nothing like dynamic tests which don't correspond
directly to a java class and method name,
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates XML report content in
surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it (eg
Jenkins JUnit plugin).
* should surefire uses be forced to enable phrased naming so that reporting
tools can access the dynamic folder and test names?
* how can XML reporting tools show the nested structure of the DynamicFolders?
** can XML reporting tools easily tell whether the XML was generated with
phrased or non phrased settings? Should they need to?
** if using phrased, should they parse the phrased name attribute to find the
dynamic folder and test names?
** if using non phrased, is there a way of surefire safely injecting the
dynamic folder and test names into an existing attribute which will work in a
backwards compatible way with the XML reporting tools?
Also the internal ordering of the phrased name attribute seems inconsistent, or
at least I don't understand it. Since "test" is inside "folder1", surely
"folder1" and "test" should be next to each other rather than at opposite ends
of the value? If we expect ordering by heirarchy position then we would expect
the folder/method names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
supported as XML report format and at that point, I guess support for dynamic
tests would be easy to add. However that could be years off, if at all.
Anyway, back to the real world, Surefire using the old ant XSD can generate two
variants of the XML report, using non phrased or phrased names. Being able to
control the phrased and non phrased attributes individually seems to make no
sense as the attributes are related. But anyway, we have to make do.
Seems for JUnit5 XML reporting, the intention for the "name" attribute value
was:
* non phrased is the TestIdentifier.legacyReportingName
* phrased is the TestIdentifier.displayName
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
The JUnit5 legacyReportingName for the example looks like
"createDynamicTests[1][1][1]". However this is basically no use as it uses
synthetic ids rather than the real names and will end up as a flat list without
any hierarchical structure in XML reporting tools. From memory. this was
actually used in earlier surefire
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
However, things are not as good when using surefire as it still uses the years
old ant XSD for XML reporting and that XSD doesn't support anything apart from
basic Java usage, certainly nothing like dynamic tests which don't correspond
directly to a java class and method name,
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates XML report content in
surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it (eg
Jenkins JUnit plugin).
* should surefire uses be forced to enable phrased naming so that reporting
tools can access the dynamic folder and test names?
* how can XML reporting tools show the nested structure of the DynamicFolders?
** can XML reporting tools easily tell whether the XML was generated with
phrased or non phrased settings? Should they need to?
** if using phrased, should they parse the phrased name attribute to find the
dynamic folder and test names?
** if using non phrased, is there a way of surefire safely injecting the
dynamic folder and test names into an existing attribute which will work in a
backwards compatible way with the XML reporting tools?
Also the internal ordering of the phrased name attribute seems inconsistent, or
at least I don't understand it. Since "test" is inside "folder1", surely
"folder1" and "test" should be next to each other rather than at opposite ends
of the value? If we expect ordering by heirarchy position then we would expect
the folder/method names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
supported as XML report format and at that point, I guess support for dynamic
tests would be easy to add. However that could be years off, if at all.
Anyway, back to the real world, Surefire using the old ant XSD has two variants
of the XML report, using non phrased method names, or phrased method names.
Dynamic tests don't have a non phrased name. So its not really clear to me what
the distinction between phrased and non phrased names means for Dynamic tests.
Nevertheless, phrased and non phrased is what Surefire is designed for, so we
have to come up with something.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
JUnit5 provides a legacyReportingName e.g. "createDynamicTests[1][1][1]" which
I think was intended to use in the name attribute of the old XML report format.
However this is basically no use as it uses synthetic ids rather than the real
names and will end up as a flat list without any hierarchical structure in XML
reporting tools. I think this was actually used in earlier surefire
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
However, things are not as good when using surefire as it still uses the years
old ant XSD for XML reporting and that XSD doesn't support anything apart from
basic Java usage, certainly nothing like dynamic tests which don't correspond
directly to a java class and method name,
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates XML report content in
surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it (eg
Jenkins JUnit plugin).
* should surefire uses be forced to enable phrased naming so that reporting
tools can access the dynamic folder and test names?
* how can XML reporting tools show the nested structure of the DynamicFolders?
** can XML reporting tools easily tell whether the XML was generated with
phrased or non phrased settings? Should they need to?
** if using phrased, should they parse the phrased name attribute to find the
dynamic folder and test names?
** if using non phrased, is there a way of surefire safely injecting the
dynamic folder and test names into an existing attribute which will work in a
backwards compatible way with the XML reporting tools?
Also the internal ordering of the phrased name attribute seems inconsistent, or
at least I don't understand it. Since "test" is inside "folder1", surely
"folder1" and "test" should be next to each other rather than at opposite ends
of the value? If we expect ordering by heirarchy position then we would expect
the folder/method names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
supported as XML report format and at that point, I guess support for dynamic
tests would be easy to add. However that could be years off, if at all.
Anyway, back to the real world, Surefire using the old ant XSD has two variants
of the XML report, using non phrased method names, or phrased method names.
Dynamic tests don't have a non phrased name. So its not really clear to me what
the distinction between phrased and non phrased names means for Dynamic tests.
Nevertheless, phrased and non phrased is what Surefire is designed for, so we
have to come up with something.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815095#comment-17815095
]
Geoff Soutter commented on SUREFIRE-2147:
-
[~tibordigana] [~marcphilipp] Are you able to provide any design guidance on
this?
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> However, things are not as good when using surefire as it still uses the
> years old ant XSD for XML reporting and that XSD doesn't support anything
> apart from basic Java usage, certainly nothing like dynamic tests which don't
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates XML report content in
> surefire-reports like so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the
> DynamicFolders?
> ** can XML reporting tools easily tell whether the XML was generated with
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the
> dynamic folder and test names into an existing attribute which will work in a
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent,
> or at least I don't understand the ordering. Surely folder1 should come
> before folder2 If we expect ordering by heirarchy position then we would
> expect the folder/method names like "createDynamicTest() folder1 folder2
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
> supported as XML report format and at that point, I guess support for dynamic
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD has two
> variants of the XML report, using non phrased method names, or phrased method
> names.
> Dynamic tests don't have a non phrased name. So its not really
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
However, things are not as good when using surefire as it still uses the years
old ant XSD for XML reporting and that XSD doesn't support anything apart from
basic Java usage, certainly nothing like dynamic tests which don't correspond
directly to a java class and method name,
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates XML report content in
surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it (eg
Jenkins JUnit plugin).
* should surefire uses be forced to enable phrased naming so that reporting
tools can access the dynamic folder and test names?
* how can XML reporting tools show the nested structure of the DynamicFolders?
** can XML reporting tools easily tell whether the XML was generated with
phrased or non phrased settings? Should they need to?
** if using phrased, should they parse the phrased name attribute to find the
dynamic folder and test names?
** if using non phrased, is there a way of surefire safely injecting the
dynamic folder and test names into an existing attribute which will work in a
backwards compatible way with the XML reporting tools?
Also the internal ordering of the phrased name attribute seems inconsistent, or
at least I don't understand the ordering. Surely folder1 should come before
folder2 If we expect ordering by heirarchy position then we would expect the
folder/method names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
supported as XML report format and at that point, I guess support for dynamic
tests would be easy to add. However that could be years off, if at all.
Anyway, back to the real world, Surefire using the old ant XSD has two variants
of the XML report, using non phrased method names, or phrased method names.
Dynamic tests don't have a non phrased name. So its not really clear to me what
the distinction between phrased and non phrased names means for Dynamic tests.
Nevertheless, phrased and non phrased is what Surefire is designed for, so we
have to come up with something.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
However, things are not as good when using surefire as it still uses the years
old ant XSD for XML reporting and that XSD doesn't support anything apart from
basic Java usage, certainly nothing like dynamic tests which don't correspond
directly to a java class and method name,
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates XML report content in
surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it?
* should we be forced to enable phrased naming so that reporting tools can
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure
of the DynamicFolders? Should they parse the phrased method name to find the
individual components?
Also the internal ordering seems inconsistent, or at least I don't understand
the structure. surely folder1 should come before folder2 If we expect the
ordering defined by heirarchy position then we would expect the folder/method
names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
supported as XML report format and at that point, I guess support for dynamic
tests would be easy to add. However that could be years off, if at all.
Anyway, back to the real world, Surefire using the old ant XSD has two variants
of the XML report, using non phrased method names, or phrased method names.
Dynamic tests don't have a non phrased name. So its not really clear to me what
the distinction between phrased and non phrased names means for Dynamic tests.
Nevertheless, phrased and non phrased is what Surefire is designed for, so we
have to come up with something.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
h3. Phrased
For phrased usage, its less obvious what to do.
The nested structure is injected into a single XML attribute and I haven't seen
any documentation of the internal structure of that attribute.
The ordering looks inconsistent as it is and possibly needs fixing. However,
changing this value without any schema as to
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
However, things are not as good when using surefire as it still uses the years
old ant XSD for XML reporting and that XSD doesn't support anything apart from
basic Java usage, certainly nothing like dynamic tests which don't correspond
directly to a java class and method name,
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates XML report content in
surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it?
* should we be forced to enable phrased naming so that reporting tools can
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure
of the DynamicFolders? Should they parse the phrased method name to find the
individual components?
Also the internal ordering seems inconsistent, or at least I don't understand
the structure. surely folder1 should come before folder2 If we expect the
ordering defined by heirarchy position then we would expect the folder/method
names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
We have two ways to look at this, using non phrased method names, or phrased
method names.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
h3. Phrased
For phrased usage, its less obvious what to do.
The nested structure is injected into a single XML attribute and I haven't seen
any documentation of the internal structure of that attribute.
The ordering looks inconsistent as it is and possibly needs fixing. However,
changing this value without any schema as to what the contents mean seems
dangerous.
Casual testing with an available Jenkins instance seems to indicate that
Jenkins JUnit already has some capability to parse the phrased method name as
it shows folder2 as a "package" (but folder1 is lost).
To get an overall useful result here I guess we'd need to
* create and publish some kind of docs / schema on the internal structure we
want to support for downstream reporting tools
* adjust whatever output we have now to meet the docs/schema, if required
* wait for downstream reporting tools to support it
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Priority: Major (was: Minor)
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates content in surefire-reports like
> so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it?
> * should we be forced to enable phrased naming so that reporting tools can
> access the dynamic folder and test names?
> * assuming the above is true, how can reporting tools show the nested
> structure of the DynamicFolders? Should they parse the phrased method name to
> find the individual components?
> Also the internal ordering seems inconsistent, or at least I don't understand
> the structure. surely folder1 should come before folder2 If we expect the
> ordering defined by heirarchy position then we would expect the folder/method
> names like "createDynamicTest() folder1 folder2 test1".
> h3. Solution Ideas
> We have two ways to look at this, using non phrased method names, or phrased
> method names.
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So
> anything we do here would be an improvement.
> So for non phrased method names, we could add the dynamic names as synthetic
> child packages in the classname attribute.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
>classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> h3. Phrased
> For phrased usage, its less obvious what to do.
> The nested structure is injected into a single XML attribute and I haven't
> seen any documentation of the internal structure of that attribute.
> The ordering looks inconsistent as it is and possibly needs fixing. However,
> changing
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Priority: Minor (was: Major)
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Minor
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates content in surefire-reports like
> so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it?
> * should we be forced to enable phrased naming so that reporting tools can
> access the dynamic folder and test names?
> * assuming the above is true, how can reporting tools show the nested
> structure of the DynamicFolders? Should they parse the phrased method name to
> find the individual components?
> Also the internal ordering seems inconsistent, or at least I don't understand
> the structure. surely folder1 should come before folder2 If we expect the
> ordering defined by heirarchy position then we would expect the folder/method
> names like "createDynamicTest() folder1 folder2 test1".
> h3. Solution Ideas
> We have two ways to look at this, using non phrased method names, or phrased
> method names.
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So
> anything we do here would be an improvement.
> So for non phrased method names, we could add the dynamic names as synthetic
> child packages in the classname attribute.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
>classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> h3. Phrased
> For phrased usage, its less obvious what to do.
> The nested structure is injected into a single XML attribute and I haven't
> seen any documentation of the internal structure of that attribute.
> The ordering looks inconsistent as it is and possibly needs fixing. However,
> changing
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Affects Version/s: 3.2.5
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7, 3.2.5
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection createDynamicTests() {
> List folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> folder2
> * test1
> * test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates content in surefire-reports like
> so
> {code}
>time="0.012"/>
>time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
>classname="pkg.JUnit5DynamicTest" time="0.011"/>
>classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it?
> * should we be forced to enable phrased naming so that reporting tools can
> access the dynamic folder and test names?
> * assuming the above is true, how can reporting tools show the nested
> structure of the DynamicFolders? Should they parse the phrased method name to
> find the individual components?
> Also the internal ordering seems inconsistent, or at least I don't understand
> the structure. surely folder1 should come before folder2 If we expect the
> ordering defined by heirarchy position then we would expect the folder/method
> names like "createDynamicTest() folder1 folder2 test1".
> h3. Solution Ideas
> We have two ways to look at this, using non phrased method names, or phrased
> method names.
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So
> anything we do here would be an improvement.
> So for non phrased method names, we could add the dynamic names as synthetic
> child packages in the classname attribute.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
>classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> h3. Phrased
> For phrased usage, its less obvious what to do.
> The nested structure is injected into a single XML attribute and I haven't
> seen any documentation of the internal structure of that attribute.
> The ordering looks inconsistent as it is and possibly needs fixing. However,
> changing this
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates content in surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it?
* should we be forced to enable phrased naming so that reporting tools can
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure
of the DynamicFolders? Should they parse the phrased method name to find the
individual components?
Also the internal ordering seems inconsistent, or at least I don't understand
the structure. surely folder1 should come before folder2 If we expect the
ordering defined by heirarchy position then we would expect the folder/method
names like "createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
We have two ways to look at this, using non phrased method names, or phrased
method names.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
h3. Phrased
For phrased usage, its less obvious what to do.
The nested structure is injected into a single XML attribute and I haven't seen
any documentation of the internal structure of that attribute.
The ordering looks inconsistent as it is and possibly needs fixing. However,
changing this value without any schema as to what the contents mean seems
dangerous.
Casual testing with an available Jenkins instance seems to indicate that
Jenkins JUnit already has some capability to parse the phrased method name as
it shows folder2 as a "package" (but folder1 is lost).
To get an overall useful result here I guess we'd need to
* create and publish some kind of docs / schema on the internal structure we
want to support for downstream reporting tools
* adjust whatever output we have now to meet the docs/schema, if required
* wait for downstream reporting tools to support it
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates content in surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it?
* should we be forced to enable phrased naming so that reporting tools can
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure
of the DynamicFolders? Should they parse the phrased method name to find the
individual components?
Also the internal ordering seems inconsistent, if we expect the ordering
defined by heirarchy position then we would expect the folder/method names like
"createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
We have two ways to look at this, using non phrased method names, or phrased
method names.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
h3. Phrased
For phrased usage, its less obvious what to do.
The nested structure is injected into a single XML attribute and I haven't seen
any documentation of the internal structure of that attribute.
The ordering looks inconsistent as it is and possibly needs fixing. However,
changing this value without any schema as to what the contents mean seems to be
dangerous.
Casual testing with an available Jenkins instance seems to indicate that
Jenkins JUnit already has some capability to parse the phrased method name as
it shows folder2 as a "package" (but folder1 is lost).
To get an overall useful result here I guess we'd need to
* create and publish some kind of docs / schema on the internal structure we
want to support for downstream reporting tools
* adjust whatever output we have now to meet the docs/schema, if required
* wait for downstream reporting tools to support it
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates content in surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it?
* should we be forced to enable phrased naming so that reporting tools can
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure
of the DynamicFolders? Should they parse the phrased method name to find the
individual components?
Also the internal ordering seems inconsistent, if we expect the ordering
defined by heirarchy position then we would expect the folder/method names like
"createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
We have two ways to look at this, using non phrased method names, or phrased
method names.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
h3. Phrased
For phrased usage, its less obvious what to do. The nested structure is
injected into a single XML attribute and I haven't seen any documentation of
the internal structure of that attribute. The ordering looks inconsistent as it
is and possibly needs fixing. However, changing this value without any schema
as to what the contents mean seems to be dangerous.
Casual testing with an available Jenkins instance seems to indicate that
Jenkins JUnit already has some capability to parse the phrased method name as
it shows folder2 as a "package" (but folder1 is lost).
Getting an overall useful result here would mostly rely on fixes in the
downstream reporting tools.
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* JUnit5DynamicTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates content in surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report, with the DynamicContainer and DynamicTest names
injected into the phrased "method" name.
>From the perspective of someone with a lot of dynamic tests in a huge folder
>structure, this is less than ideal as the nested structure is not exposed in
>the test report in a clean structure way, as it was with package names.
So how do the tools that process this XML into HTML reports handle it?
* should we be forced to enable phrased naming so that reporting tools can
access the dynamic folder and test names?
* assuming the above is true, how can reporting tools show the nested structure
of the DynamicFolders? Should they parse the phrased method name to find the
individual components?
Also the internal ordering seems inconsistent, if we expect the ordering
defined by heirarchy position then we would expect the folder/method names like
"createDynamicTest() folder1 folder2 test1".
h3. Solution Ideas
We have two ways to look at this, using non phrased method names, or phrased
method names.
h3. Non Phrased
Dynamic tests with non phrased method names are currently not usable. So
anything we do here would be an improvement.
So for non phrased method names, we could add the dynamic names as synthetic
child packages in the classname attribute.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
h3. Phrased
For phrased usage, its less obvious what to do. The nested structure is
injected into a single XML attribute and I haven't seen any documentation of
the internal structure of that attribute. The ordering looks inconsistent as it
is and possibly needs fixing. However, changing this value without any schema
as to what the contents mean seems to be dangerous.
Casual testing with an available Jenkins instance seems to indicate that
Jenkins JUnit already has some capability to parse the phrased method name as
it shows folder2 as a "package" (but folder1 is lost).
Getting an overall useful result here would mostly rely on fixes in the
downstream reporting tools.
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
import ...
public class JUnit5DynamicTest {
@TestFactory
Collection createDynamicTests() {
List folder2 = new ArrayList<>();
folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List folder1 = new ArrayList<>();
folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
List rootFolder = new ArrayList<>();
rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
return rootFolder;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest (pkg)
** createDynamicTests()
*** folder1
folder2
* test1
* test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
reported configuration disabled, it creates content in surefire-reports like so
{code}
{code}
In this case the Dynamic Tests are completely obscured, instead we get
duplicate reports for the @TestFactory method.
With the example "phrased" reporter configuration "use" properties enabled, we
get
{code:java}
{code}
Here all the Dynamic tests are created as synthetic "methods" of the physical
class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the phrased "method" name. Also the internal ordering seems
inconsistent, if we expect the ordering defined by heirarchy position then we
would expect the folder/method names like "createDynamicTest() folder1 folder2
test1".
Problem is how do the tools that process this XML into HTML reports handle it?
* should we enable phrased naming to expose the
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
Or if we considered the java package is a distraction when using dynamic tests
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package pkg;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
Or if we considered the java package is a distraction when using dynamic tests
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the
[jira] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147 ]
Geoff Soutter deleted comment on SUREFIRE-2147:
-
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved. Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
The Jenkins JUnit plugin now creates an hierarchical report. Saying that, it is
not ideal in Jenkins, as it lacks the synthetic "Test" name of this proposal.
This means there is no consistent mapping from container to package - the
"last" container is mapped to class, so
* (negative) on the TestReport page the last container "disappears" from the
package listing,
* (positive) there is no need to "click through" the synthetic "Tests" class to
see results.
TBH neither way is perfect, it is only a matter of choosing the least worst
option, but for my purpose having Containers mapping cleanly to Packages is
important.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
I latest tested in 3.0.0-M7. Not sure exactly what fixed it, but these look
related
* M8 - SUREFIRE-2117
* M9 - SUREFIRE-2065
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
> {code}
> Or if we considered the java package is a distraction when using dynamic tests
> {code:java}
>
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[ https://issues.apache.org/jira/browse/SUREFIRE-2147 ]
Geoff Soutter deleted comment on SUREFIRE-2147:
-
was (Author: gsoutter):
Had a quick look at 3.2.5 to see if any changes.
Not sure if I noticed before, but tried 3.2.5 on a real use case, which maps a
folder structure into Dynamic tests. Therefore I have parent Containers nested
inside child Containers. The current implementation seems to throw away the
parent containers, only the leaf container is shown in the report.
I think I can live with the rest of the behaviour, but throwing away "parent"
container names fundamentally breaks my use case.
Hopefully I can set aside some time to do contribute a PR for this (although
I've promised that a few times already and yet to deliver anything. Sigh).
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
> {code}
> Or if we considered the java package is a distraction when using dynamic tests
> {code:java}
>
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808437#comment-17808437
]
Geoff Soutter commented on SUREFIRE-2147:
-
Had a quick look at 3.2.5 to see if any changes.
Not sure if I noticed before, but tried 3.2.5 on a real use case, which maps a
folder structure into Dynamic tests. Therefore I have parent Containers nested
inside child Containers. The current implementation seems to throw away the
parent containers, only the leaf container is shown in the report.
I think I can live with the rest of the behaviour, but throwing away "parent"
container names fundamentally breaks my use case.
Hopefully I can set aside some time to do contribute a PR for this (although
I've promised that a few times already and yet to deliver anything. Sigh).
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Tests"/>
> {code}
> Or if we considered the java package is a distraction when using dynamic tests
> {code:java}
>
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 2:09 AM:
---
Checked surefire 3.2.2 and seems this has improved. Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
The Jenkins JUnit plugin now creates an hierarchical report. Saying that, it is
not ideal in Jenkins, as it lacks the synthetic "Test" name of this proposal.
This means there is no consistent mapping from container to package - the
"last" container is mapped to class, so
* (negative) on the TestReport page the last container "disappears" from the
package listing,
* (positive) there is no need to "click through" the synthetic "Tests" class to
see results.
TBH neither way is perfect, it is only a matter of choosing the least worst
option, but for my purpose having Containers mapping cleanly to Packages is
important.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
I latest tested in 3.0.0-M7. Not sure exactly what fixed it, but these look
related
* M8 - SUREFIRE-2117
* M9 - SUREFIRE-2065
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved. Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
The Jenkins JUnit plugin now creates an hierarchical report. Saying that, it is
not ideal in Jenkins, as it lacks the synthetic "Test" name of this proposal.
This means there is no consistent mapping from container to package - the
"last" container is mapped to class, so
* (negative) on the TestReport page the last container "disappears" from the
package listing,
* (positive) there is no need to "click through" the synthetic "Tests" class to
see results.
TBH neither way is perfect, it is only a matter of choosing the least worst
option, but for my purpose having Containers mapping cleanly to Packages is
important.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:37 AM:
---
Checked surefire 3.2.2 and seems this has improved. Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
The Jenkins JUnit plugin now creates an hierarchical report. Saying that, it is
not ideal in Jenkins, as it lacks the synthetic "Test" name of this proposal.
This means there is no consistent mapping from container to package - the
"last" container is mapped to class, so
* (negative) on the TestReport page the last container "disappears" from the
package listing,
* (positive) there is no need to "click through" the synthetic "Tests" class to
see results.
TBH neither way is perfect, it is only a matter of choosing the least worst
option, but for my purpose having Containers mapping cleanly to Packages is
important.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved. Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
The Jenkins JUnit plugin now creates an hierarchical report. Saying that, it is
not ideal in Jenkins, as it lacks the synthetic "Test" name of this proposal.
This means there is no consistent mapping from container to package - the
"last" container is mapped to class, so
* (negative) on the TestReport page the last container "disappears" from the
package listing,
* (positive) there is no need to "click through" the synthetic Test to see
results.
TBH neither way is perfect, it is only a matter of choosing the least worst
option, but for my purpose having Containers mapping cleanly to Packages is
important.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:36 AM:
---
Checked surefire 3.2.2 and seems this has improved. Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
The Jenkins JUnit plugin now creates an hierarchical report. Saying that, it is
not ideal in Jenkins, as it lacks the synthetic "Test" name of this proposal.
This means there is no consistent mapping from container to package - the
"last" container is mapped to class, so
* (negative) on the TestReport page the last container "disappears" from the
package listing,
* (positive) there is no need to "click through" the synthetic Test to see
results.
TBH neither way is perfect, it is only a matter of choosing the least worst
option, but for my purpose having Containers mapping cleanly to Packages is
important.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:31 AM:
---
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
>
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
Or if we considered the java package is a distraction when using dynamic tests
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:30 AM:
---
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
>
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Tests) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
Or if we considered the java package irrelevant in this situation
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:25 AM:
---
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing, but on the plus
side there is no need to "click through" the synthetic Test to see results. TBH
neither way is perfect, it is only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing. TBH neither way
is perfect, its only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
[jira] [Comment Edited] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter edited comment on SUREFIRE-2147 at 11/22/23 1:23 AM:
---
Checked surefire 3.2.2 and seems this has improved.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
Saying that, it is not ideal in Jenkins, as it lacks the synthetic "Test" name
of this proposal, which means there is no consistent mapping from container to
package - the "last" container is mapped to class. This means on the TestReport
page the last container "disappears" from the package listing. TBH neither way
is perfect, its only a matter of choosing the least worst option.
Also, in case of a Test not having a Container, it is inconsistent.
{code}
{code}
Not quite sure what changed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
was (Author: gsoutter):
Checked surefire 3.2.2 and seems this has been mostly fixed.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
In case of test being in the root folder, it is inconsistent. But I suspect few
would have that scenario,
{code}
{code}
Not quite sure what fixed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
[jira] [Commented] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788579#comment-17788579
]
Geoff Soutter commented on SUREFIRE-2147:
-
Checked surefire 3.2.2 and seems this has been mostly fixed.
Whereas before we get this
{code}
{code}
Looks like now we get
{code}
{code}
This is a big improvement, and it seems Jenkins JUnit plugin can parse the XML
and create a hierarchical report.
In case of test being in the root folder, it is inconsistent. But I suspect few
would have that scenario,
{code}
{code}
Not quite sure what fixed it, but I suspect changes for SUREFIRE-2117, which
was done in 3.0.0-M8, and I last tested on 3.0.0-M7. Sigh. If so, thanks
[~sjaranowski].
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Affects Version/s: 3.0.0-M7
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M7
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as synthetic "methods" of a
> single class in the HTML report. The DynamicContainer and DynamicTest names
> are injected into the synthetic "method" name. That is, from a reporting
> perspective, they are created as a flat list (also it seems the folder name
> is in the wrong place, it seems it should be on the left of the method name,
> eg "createDynamicTests() aFolder test1" in order for the report to match the
> internal structure).
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686136#comment-17686136
]
Geoff Soutter commented on SUREFIRE-2144:
-
JUnit project suggested a workaround in
https://github.com/junit-team/junit4/issues/1758, of catching the suite()
exception internally. Using 20/20 hindsight, this approach is quite obvious.
Sigh. It might explain why this issue has not yet been fixed.
I combined catching the suite() Exception and returning an empty Suite with
. With Surefure, this achieved the goal of ensuring suite
Exceptions do not corrupt the Jenkins Unit test history reporting. However, in
Failsafe, doesn't work with . It seems
there was an old bug which is claimed fixed but has not been, see SUREFIRE-1024.
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.net.ConnectException: Connection refused (Connection
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680131#comment-17680131
]
Geoff Soutter edited comment on SUREFIRE-2144 at 2/1/23 5:52 AM:
-
OK, so I made a local version of JUnit to explore what a JUnit fix might looks
like. This has ErrorRunningReporter replaced with throw
{{RuntimeException("PREVENTING ...". e)}} in both {{SuiteMethodBuilder}} and
{{{}AllDefaultPossibilitiesBuilder{}}}. When run directly from IDEA, it reports
{{{}"No tests were found"{}}}, which is what we want - no fake tests created.
But when tested with Surefire, slightly different fake test is synthesised. In
this case, we can see the {{initializationFailed}} disappears which makes me
think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:59)
... 9 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection
refused (Connection refused)
{code}
So it seems fixing this problem would require changes in both JUnit and
Surefire. Sigh.
was (Author: gsoutter):
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}, which is what we
want - no fake tests created.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680109#comment-17680109
]
Geoff Soutter edited comment on SUREFIRE-2144 at 2/1/23 5:51 AM:
-
Did a bit more static analysis and checked the JUnit 4 code from github.
There in RunnerBuilder, it does this
{code}
public Runner safeRunnerForClass(Class testClass) {
try {
Runner runner = runnerForClass(testClass);
if (runner != null) {
configureRunner(runner);
}
return runner;
} catch (Throwable e) {
return new ErrorReportingRunner(testClass, e);
}
}
{code}
ErrorReportingRunner is the thing that synthesizes the fake tests.
{code}
private Description describeCause() {
return Description.createTestDescription(classNames,
"initializationError");
}
{code}
The issue is that SuiteMethodBuilder overrides RunnerBuilder, but doesn't
override safeRunnerForClass, which provides behaviour which is not safe for
Suites (i.e. generated fake fails in place of the actual generated tests).
So from my static analysis, seems this is a JUnit issue rather than Surefire
issue.
I raised https://github.com/junit-team/junit4/issues/1758 for that.
was (Author: gsoutter):
Did a bit more static analysis and checked the JUnit 4 code from github.
There in RunnerBuilder, it does this
{code}
public Runner safeRunnerForClass(Class testClass) {
try {
Runner runner = runnerForClass(testClass);
if (runner != null) {
configureRunner(runner);
}
return runner;
} catch (Throwable e) {
return new ErrorReportingRunner(testClass, e);
}
}
{code}
ErrorReportingRunner is the thing that synthesizes the fake tests.
{code}
private Description describeCause() {
return Description.createTestDescription(classNames,
"initializationError");
}
{code}
The issue is that SuiteMethodBuilder overrides RunnerBuilder, but doesn't
override safeRunnerForClass, which provides behaviour which is not safe for
Suites (i.e. generated fake fails in place of the actual generated tests).
So from my static analysis, seems this is a JUnit issue rather than Surefire
issue.
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as synthetic "methods" of a
single class in the HTML report. The DynamicContainer and DynamicTest names are
injected into the synthetic "method" name. That is, from a reporting
perspective, they are created as a flat list (also it seems the folder name is
in the wrong place, it seems it should be on the left of the method name, eg
"createDynamicTests() aFolder test1" in order for the report to match the
internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list (also it
seems the folder name is in the wrong place, it seems it should be on the left
of the method name, eg "createDynamicTests() aFolder test1" in order for the
report to match the internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list (also it
seems the folder name is in the wrong place, it seems it should be on the left
of the method name, eg "createDynamicTests() aFolder test1" in order for the
report to match the internal structure).
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following hierarchical structure in the UI
treeview
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could
[jira] [Commented] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681251#comment-17681251
]
Geoff Soutter commented on SUREFIRE-2144:
-
I created SUREFURE-2147 to try and make the equivalent JUnit5 dynamically
created tests reporting work at least as well as JUnit 4 does.
Hopefully this might be an escape hatch for the above ancient JUnit4 issue.
Still I guess I need to test the behaviour when the test method fails before
any tests are created
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> ...
> ... 23 more
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] MySuite.suite:xx » Runtime java.net.ConnectException: Connection
> refused (...
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
>
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681251#comment-17681251
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/27/23 9:28 AM:
--
I created SUREFIRE-2147 to try and make the equivalent JUnit5 dynamically
created tests reporting work at least as well as JUnit 4 does.
Hopefully this might be an escape hatch for the above ancient JUnit4 issue.
Still I guess I need to test the behaviour when the test method fails before
any tests are created
was (Author: gsoutter):
I created SUREFURE-2147 to try and make the equivalent JUnit5 dynamically
created tests reporting work at least as well as JUnit 4 does.
Hopefully this might be an escape hatch for the above ancient JUnit4 issue.
Still I guess I need to test the behaviour when the test method fails before
any tests are created
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Summary: Report for JUnit5 DynamicContainer/DynamicTest loses the
hierarchical structure (was: Report for JUnit5 DynamicContainer/DynamicTest
doesn't preserve the hierarchical structure)
> Report for JUnit5 DynamicContainer/DynamicTest loses the hierarchical
> structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following structure
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as the child of a single
> class in the HTML report. The folder name is injected into the "method" name.
> That is, from a reporting perspective, they are created as a flat list.
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2147) JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Summary: JUnit5 DynamicContainer/DynamicTest report loses the hierarchical
structure (was: Report for JUnit5 DynamicContainer/DynamicTest loses the
hierarchical structure)
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following structure
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as the child of a single
> class in the HTML report. The folder name is injected into the "method" name.
> That is, from a reporting perspective, they are created as a flat list.
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the hierarchical structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Summary: Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve
the hierarchical structure (was: Report for JUnit5
DynamicContainer/DynamicTest doesn't preserve the nested structure)
> Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the
> hierarchical structure
> --
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following structure
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as the child of a single
> class in the HTML report. The folder name is injected into the "method" name.
> That is, from a reporting perspective, they are created as a flat list.
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get a breakdown on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests using dynamic containers
> should have those container names structured as synthetic packages. This
> allows us the benefits of having the nested structure modelled correctly in
> the HTML report.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | simple class name | static placeholder (e.g. Test) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the nested structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get a breakdown on any
particular DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could do that by moving the method name and
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the nested structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests using dynamic containers should
have those container names structured as synthetic packages. This allows us the
benefits of having the nested structure modelled correctly in the HTML report.
That is, when mapping from DynamicTests to Java packages, we have:
|| Java Structure || Dynamic Test Structure ||
| package identifier | DynamicContainer name |
| simple class name | static placeholder (e.g. Test) |
| method name | DynamicTest name |
I think we could do that by moving the method name and folder name from name=
into classname= and adding a static placeholder for the classname.
{code:java}
{code}
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code:java}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name. This is acts as a placeholder for
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the nested structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code:java}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name. This is acts as a placeholder for Classname as
DynamicTests don't have a "class" concept.
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code:java}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name. This is acts as a placeholder for Class as DynamicTests
don't have a "class" concept.
> Report for
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the nested structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code:java}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name. This is acts as a placeholder for Class as DynamicTests
don't have a "class" concept.
was:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code:java}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
> Report for JUnit5 DynamicContainer/DynamicTest doesn't
[jira] [Updated] (SUREFIRE-2147) Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the nested structure
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Summary: Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve
the nested structure (was: Surefire report for JUnit5
DynamicContainer/DynamicTest doesn't work with HTML generation)
> Report for JUnit5 DynamicContainer/DynamicTest doesn't preserve the nested
> structure
>
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
>Reporter: Geoff Soutter
>Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package samplepackage;
> ...
> public class SampleTest {
> @TestFactory
> Collection createDynamicTests() {
> List testList = new ArrayList<>();
> testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List root = new ArrayList<>();
> root.add(DynamicContainer.dynamicContainer("aFolder", testList));
> return root;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following structure
> * SampleTest
> ** createDynamicTests()
> *** aFolder
> test1
> test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
> enabled, it creates contents in surefire-reports like so:
> {code:java}
>classname="samplepackage.SampleTest" time="0.02"/>
>classname="samplepackage.SampleTest" time="0.001"/>
> {code}
> Here all the nested synthetic tests are created as the child of a single
> class in the HTML report. The folder name is injected into the "method" name.
> That is, from a reporting perspective, they are created as a flat list.
> This doesn't scale nicely as the number of tests in the dynamically created
> "folder" structure grows. We can't view the different DynamicContainers
> separately in the rendered HTML reports. We can't get history on any
> particular DynamicContainer in the rendered HTML reports.
> h3. Solution Ideas
> I think what we want here is that dynamic tests in folders should have those
> folders structured as synthetic packages. This allows us the benefits of
> having the nested structure modelled correctly in the HTML report. I think we
> could do that by moving the method name and folder name from name= into
> classname=
> {code:java}
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.02"/>
>classname="samplepackage.SampleTest.createDynamicTests().aFolder.Test"
> time="0.001"/>
> {code}
> A compromise I have made here is that I have injected ".Test" at the end as
> it allows the entire DynamicContainer "folder" structure to be visible in the
> virtual package name - DynamicTests don't have a "class" concept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2147) Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with HTML generation
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
h3. Description of the issue
I created a simple example of JUnit5 DynamicContainer/DynamicTest
{code:java}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code:java}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
h3. Solution Ideas
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code:java}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
was:
I created a really simple example of JUnit5 DynamicContainer/DynamicTest
{code}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
> Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with
> HTML generation
>
[jira] [Updated] (SUREFIRE-2147) Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with HTML generation
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
I created a really simple example of JUnit5 DynamicContainer/DynamicTest
{code}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
{code}
{code}
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
was:
I created a really simple example of JUnit5 DynamicContainer/DynamicTest
{code}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
> Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with
> HTML generation
> -
>
> Key:
[jira] [Updated] (SUREFIRE-2147) Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with HTML generation
[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2147:
Description:
I created a really simple example of JUnit5 DynamicContainer/DynamicTest
{code}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, with the example "phrased" reporter configuration
enabled, it creates contents in surefire-reports like so:
{code}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
A compromise I have made here is that I have injected ".Test" at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
was:
I created a really simple example of JUnit5 DynamicContainer/DynamicTest
{code}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, it creates contents in surefire-reports like so:
{code}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
A compromise I have made here is that I have injected .Test at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
> Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with
> HTML generation
> -
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
[jira] [Created] (SUREFIRE-2147) Surefire report for JUnit5 DynamicContainer/DynamicTest doesn't work with HTML generation
Geoff Soutter created SUREFIRE-2147:
---
Summary: Surefire report for JUnit5 DynamicContainer/DynamicTest
doesn't work with HTML generation
Key: SUREFIRE-2147
URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
Project: Maven Surefire
Issue Type: Bug
Reporter: Geoff Soutter
I created a really simple example of JUnit5 DynamicContainer/DynamicTest
{code}
package samplepackage;
...
public class SampleTest {
@TestFactory
Collection createDynamicTests() {
List testList = new ArrayList<>();
testList.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
testList.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
List root = new ArrayList<>();
root.add(DynamicContainer.dynamicContainer("aFolder", testList));
return root;
}
private static class MyExecutable implements Executable {
@Override
public void execute() throws Throwable {
}
}
}
{code}
When I run this in IDEA. I get the following structure
* SampleTest
** createDynamicTests()
*** aFolder
test1
test2
Here we can see that dynamic tests are represented by IDEA as nested children
of the test method. We can open and close the "virtual packages" created as
usual to explore in the dynamically created test results. It is not a "flat
list".
Using surefire 3.0.0-m7, it creates contents in surefire-reports like so:
{code}
{code}
Here all the nested synthetic tests are created as the child of a single class
in the HTML report. The folder name is injected into the "method" name. That
is, from a reporting perspective, they are created as a flat list.
This doesn't scale nicely as the number of tests in the dynamically created
"folder" structure grows. We can't view the different DynamicContainers
separately in the rendered HTML reports. We can't get history on any particular
DynamicContainer in the rendered HTML reports.
I think what we want here is that dynamic tests in folders should have those
folders structured as synthetic packages. This allows us the benefits of having
the nested structure modelled correctly in the HTML report. I think we could do
that by moving the method name and folder name from name= into classname=
A compromise I have made here is that I have injected .Test at the end as it
allows the entire DynamicContainer "folder" structure to be visible in the
virtual package name - DynamicTests don't have a "class" concept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680131#comment-17680131
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 8:04 AM:
--
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}, which is what we
want - no fake tests created.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:59)
... 9 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection
refused (Connection refused)
{code}
So it seems fixing this problem would require changes in both JUnit and
Surefire. Sigh.
was (Author: gsoutter):
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
[jira] [Commented] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680131#comment-17680131
]
Geoff Soutter commented on SUREFIRE-2144:
-
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:59)
... 9 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection
refused (Connection refused)
{code}
So it seems fixing this problem would require changes in both JUnit and
Surefire. Sigh.
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection
[jira] [Commented] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680109#comment-17680109
]
Geoff Soutter commented on SUREFIRE-2144:
-
Did a bit more static analysis and checked the JUnit 4 code from github.
There in RunnerBuilder, it does this
{code}
public Runner safeRunnerForClass(Class testClass) {
try {
Runner runner = runnerForClass(testClass);
if (runner != null) {
configureRunner(runner);
}
return runner;
} catch (Throwable e) {
return new ErrorReportingRunner(testClass, e);
}
}
{code}
ErrorReportingRunner is the thing that synthesizes the fake tests.
{code}
private Description describeCause() {
return Description.createTestDescription(classNames,
"initializationError");
}
{code}
The issue is that SuiteMethodBuilder overrides RunnerBuilder, but doesn't
override safeRunnerForClass, which provides behaviour which is not safe for
Suites (i.e. generated fake fails in place of the actual generated tests).
So from my static analysis, seems this is a JUnit issue rather than Surefire
issue.
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680073#comment-17680073
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 1:57 AM:
--
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow allow some Throwables to bubble up.
One way to do that would be to add a simple feature switch (eg
synthesizeTestFailureForInitializationError=false) to control the catch
behaviour.
Another way would be to somehow detect the type of test being run and therefore
have different catch behaviour for dynamically created tests than static ones.
However, I'm not sure if surefire is aware of that, and it would be a much
bigger change.
was (Author: gsoutter):
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow allow some Throwables to bubble up.
Some ideas
* manual: a simple feature switch (eg
synthesizeTestFailureForInitializationError=false) to control the behaviour
* automatic: refactor to make the error handling appropriate for the type of
test being run
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL:
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680073#comment-17680073
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 1:51 AM:
--
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow allow some Throwables to bubble up.
Some ideas
* manual: a simple feature switch (eg
synthesizeTestFailureForInitializationError=false) to control the behaviour
* automatic: refactor to make the error handling appropriate for the type of
test being run
Does this sound plausible?
was (Author: gsoutter):
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> +
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680073#comment-17680073
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 1:42 AM:
--
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
was (Author: gsoutter):
I did some basic static analysis of the Surefire code.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test:
[jira] [Commented] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680073#comment-17680073
]
Geoff Soutter commented on SUREFIRE-2144:
-
I did some basic static analysis of the Surefire code.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at
[jira] [Updated] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Description:
My team is dynamically creating tests using the JUnit3/4 static suite method +
TestSuite API
{code}
public static junit.framework.Test suite() {
TestSuite testSuite = new TestSuite(xxx);
for (Test test: tests) {
testSuite.addTest(yyy);
}
return testSuite;
}
{code}
and then running this using Jenkins CI with Surefire.
There is a nasty failure pattern which periodically deletes the Jenkins test
history - it resets the Age of all tests in Jenkins back to 1. The history /
Age report in Jenkins is key for us as it reveals which commit caused the
failure. We definitely do not want to lose that information.
The failure pattern goes like this:
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.064
s <<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019 s
<<< ERROR!
java.lang.RuntimeException: java.net.ConnectException: Connection refused
(Connection refused)
...
at com.company.package.MySuite.suite(MySuite.java:xx)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
...
... 23 more
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] MySuite.suite:xx » Runtime java.net.ConnectException: Connection
refused (...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
It seems that the Jenkins JUnit report analysis gets completely confused when
it does the history analysis in this case. Presumably it tries to compare
* the "many" passing and failing tests from run N-1 with
* the single fake test from run N
After that I presume it decides that those many N-1 tests have been deleted,
and therefore deletes the history of those tests.
So it seems that if we value the history analysis, we need to keep the test
names stable. If so, I suspect the fix is here is that Surefire should never
create and pretend to run synthetic tests. Rather, the entire run should simply
fail - I presume throw an exception out of the surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the
[jira] [Updated] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Summary: Using JUnit4 TestSuite to create test dynamically, synthetic
initializationError failure breaks Jenkins test history (was: Synthetic
initializationError failure breaks Jenkins test history)
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> ...
> ... 23 more
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] MySuite.suite:xx » Runtime java.net.ConnectException: Connection
> refused (...
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> {code}
> There is no "initializationError" test in our source code. I presume Surefire
> has created it.
> It seems that the Jenkins JUnit report analysis gets completely confused when
> it does the history analysis in this case. Presumably it tries to
[jira] [Updated] (SUREFIRE-2144) Synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Description:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically deletes the Jenkins test
history - it resets the Age of all tests in Jenkins back to 1. The history /
Age report in Jenkins is key for us as it reveals which commit caused the
failure. We definitely do not want to lose that information.
The failure pattern goes like this:
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.064
s <<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019 s
<<< ERROR!
java.lang.RuntimeException: java.net.ConnectException: Connection refused
(Connection refused)
...
at com.company.package.MySuite.suite(MySuite.java:xx)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
...
... 23 more
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] MySuite.suite:xx » Runtime java.net.ConnectException: Connection
refused (...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
It seems that the Jenkins JUnit report analysis gets completely confused when
it does the history analysis in this case. Presumably it tries to compare
* the "many" passing and failing tests from run N-1 with
* the single fake test from run N
After that I presume it decides that those many N-1 tests have been deleted,
and therefore deletes the history of those tests.
So it seems that if we value the history analysis, we need to keep the test
names stable. If so, I suspect the fix is here is that Surefire should never
create and pretend to run synthetic tests. Rather, the entire run should simply
fail - I presume throw an exception out of the surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
was:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically deletes the Jenkins test
history - it
[jira] [Updated] (SUREFIRE-2144) Synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Summary: Synthetic initializationError failure breaks Jenkins test history
(was: Synthetic failures for initialization error breaks Jenkins test history)
> Synthetic initializationError failure breaks Jenkins test history
> -
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847
> s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046
> s <<< ERROR!
> {code}
> There is no "initializationError" test in our source code. I presume Surefire
> has created it.
> It seems that the Jenkins JUnit report analysis gets completely confused when
> it does the history analysis in this case. Presumably it tries to compare
> * the "many" passing and failing tests from run N-1 with
> * the single fake test from run N
> After that I presume it decides that those many N-1 tests have been deleted,
> and therefore deletes the history of those tests.
> So it seems that if we value the history analysis, we need to keep the test
> names stable. If so, I suspect the fix is here is that Surefire should never
> create and pretend to run synthetic tests. Rather, the entire run should
> simply fail - I presume throw an exception out of the surefire plugin back to
> maven?
> Hopefully this would then prevent Jenkins from doing the test analysis and it
> will not break the history / Age reporting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2144) Synthetic failures for initialization error breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Summary: Synthetic failures for initialization error breaks Jenkins test
history (was: Creating synthetic failures for initialization error breaks the
Jenkins test history)
> Synthetic failures for initialization error breaks Jenkins test history
> ---
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847
> s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046
> s <<< ERROR!
> {code}
> There is no "initializationError" test in our source code. I presume Surefire
> has created it.
> It seems that the Jenkins JUnit report analysis gets completely confused when
> it does the history analysis in this case. Presumably it tries to compare
> * the "many" passing and failing tests from run N-1 with
> * the single fake test from run N
> After that I presume it decides that those many N-1 tests have been deleted,
> and therefore deletes the history of those tests.
> So it seems that if we value the history analysis, we need to keep the test
> names stable. If so, I suspect the fix is here is that Surefire should never
> create and pretend to run synthetic tests. Rather, the entire run should
> simply fail - I presume throw an exception out of the surefire plugin back to
> maven?
> Hopefully this would then prevent Jenkins from doing the test analysis and it
> will not break the history / Age reporting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (SUREFIRE-2144) Creating synthetic failures for initialization error breaks the Jenkins test history
Geoff Soutter created SUREFIRE-2144:
---
Summary: Creating synthetic failures for initialization error
breaks the Jenkins test history
Key: SUREFIRE-2144
URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
Project: Maven Surefire
Issue Type: Bug
Affects Versions: 2.22.2
Reporter: Geoff Soutter
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
I think the fix is here is that Surefire should never create and pretend to run
synthetic tests, as they break the Jenkins reporting.
Rather, the entire run should fail - I presume throw an exception out of the
surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (SUREFIRE-2144) Creating synthetic failures for initialization error breaks the Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Description:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically deletes the Jenkins test
history - it resets the Age of all tests in Jenkins back to 1. The history /
Age report in Jenkins is key for us as it reveals which commit caused the
failure. We definitely do not want to lose that information.
The failure pattern goes like this:
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
It seems that the Jenkins JUnit report analysis gets completely confused when
it does the history analysis in this case. Presumably it tries to compare
* the "many" passing and failing tests from run N-1 with
* the single fake test from run N
After that I presume it decides that those many N-1 tests have been deleted,
and therefore deletes the history of those tests.
So it seems that if we value the history analysis, we need to keep the test
names stable. If so, I suspect the fix is here is that Surefire should never
create and pretend to run synthetic tests. Rather, the entire run should simply
fail - I presume throw an exception out of the surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
was:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
It seems that the Jenkins JUnit report analysis gets completely confused when
it does the history analysis in this case. Presumably it tries to compare
* the "many" passing and failing tests from run N-1 with
* the single fake test from run N
After that I presume it decides that those many N-1 tests have been deleted,
and therefore deletes the history of those tests.
So it seems that if we value the history analysis, we need to keep the test
names stable. If so, I suspect the fix is here is that Surefire should never
create and pretend to run synthetic tests. Rather, the entire run should simply
fail - I presume throw an exception out of the surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
> Creating synthetic failures for initialization error breaks the Jenkins test
> history
>
>
> Key: SUREFIRE-2144
> URL:
[jira] [Updated] (SUREFIRE-2144) Creating synthetic failures for initialization error breaks the Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Description:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
It seems that the Jenkins JUnit report analysis gets completely confused when
it does the history analysis in this case. Presumably it tries to compare
* the "many" passing and failing tests from run N-1 with
* the single fake test from run N
After that I presume it decides that those many N-1 tests have been deleted,
and therefore deletes the history of those tests.
So it seems that if we value the history analysis, we need to keep the test
names stable. If so, I suspect the fix is here is that Surefire should never
create and pretend to run synthetic tests. Rather, the entire run should simply
fail - I presume throw an exception out of the surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
was:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
I suspect the fix is here is that Surefire should never create and pretend to
run synthetic tests, as they break the Jenkins reporting.
Rather, the entire run should fail - I presume throw an exception out of the
surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
> Creating synthetic failures for initialization error breaks the Jenkins test
> history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically resets the Age of all
> tests in Jenkins back to 1. The Age report in Jenkins is key for us as it
> reveals which commit caused the failure. We don't want
[jira] [Updated] (SUREFIRE-2144) Creating synthetic failures for initialization error breaks the Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
Description:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
I suspect the fix is here is that Surefire should never create and pretend to
run synthetic tests, as they break the Jenkins reporting.
Rather, the entire run should fail - I presume throw an exception out of the
surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
was:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
I think the fix is here is that Surefire should never create and pretend to run
synthetic tests, as they break the Jenkins reporting.
Rather, the entire run should fail - I presume throw an exception out of the
surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
> Creating synthetic failures for initialization error breaks the Jenkins test
> history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically resets the Age of all
> tests in Jenkins back to 1. The Age report in Jenkins is key for us as it
> reveals which commit caused the failure. We don't want to lose that
> information.
> It goes like this
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
[jira] [Commented] (MASSEMBLY-774) too many open files
[ https://issues.apache.org/jira/browse/MASSEMBLY-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051876#comment-17051876 ] Geoff Soutter commented on MASSEMBLY-774: - OK, love being wrong, so with sample size of 1, an upgrade to 3.1 and my CI build worked. So it _maybe_ fixed. YMMV. > too many open files > --- > > Key: MASSEMBLY-774 > URL: https://issues.apache.org/jira/browse/MASSEMBLY-774 > Project: Maven Assembly Plugin > Issue Type: Bug >Affects Versions: 2.5.5 >Reporter: Dan Armbrust >Priority: Major > > I ran across this - > http://mail-archives.apache.org/mod_mbox/maven-dev/201501.mbox/%3ca2ee0d04-04e3-4c99-8842-673463862...@takari.io%3E > - and since I'm making huge zip files, I thought I would give it a try. > I configured as: > > org.apache.maven.plugins > maven-assembly-plugin > 2.5.5 > > > org.codehaus.plexus > plexus-archiver > 3.0.1 > > > org.codehaus.plexus > plexus-io > 2.6 > > > > But this lead to a failure: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (zip) on project > ... sememe/seg.3182.sememe.map (Too many open files) -> [Help 1] > I certainly could raise my ulimit: > darmbrust@overkill:/mnt/STORAGE/Work/Apelon/Workspaces/ISAAC-Core-2/va-solor-goods$ > ulimit -a > ... > open files (-n) 1024 > But it seems it would be better if the zip process limited itself to a > reasonable number of open files. I don't know the algorithm... but it seems > that double or triple the processor count would be more than enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MASSEMBLY-774) too many open files
[ https://issues.apache.org/jira/browse/MASSEMBLY-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051849#comment-17051849 ] Geoff Soutter commented on MASSEMBLY-774: - Confirmed present in 2.6. Trying 3.1 now. Suspect not fixed, since this ticket is open and no action for years. > too many open files > --- > > Key: MASSEMBLY-774 > URL: https://issues.apache.org/jira/browse/MASSEMBLY-774 > Project: Maven Assembly Plugin > Issue Type: Bug >Affects Versions: 2.5.5 >Reporter: Dan Armbrust >Priority: Major > > I ran across this - > http://mail-archives.apache.org/mod_mbox/maven-dev/201501.mbox/%3ca2ee0d04-04e3-4c99-8842-673463862...@takari.io%3E > - and since I'm making huge zip files, I thought I would give it a try. > I configured as: > > org.apache.maven.plugins > maven-assembly-plugin > 2.5.5 > > > org.codehaus.plexus > plexus-archiver > 3.0.1 > > > org.codehaus.plexus > plexus-io > 2.6 > > > > But this lead to a failure: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (zip) on project > ... sememe/seg.3182.sememe.map (Too many open files) -> [Help 1] > I certainly could raise my ulimit: > darmbrust@overkill:/mnt/STORAGE/Work/Apelon/Workspaces/ISAAC-Core-2/va-solor-goods$ > ulimit -a > ... > open files (-n) 1024 > But it seems it would be better if the zip process limited itself to a > reasonable number of open files. I don't know the algorithm... but it seems > that double or triple the processor count would be more than enough. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1546) JUnit 5 runner does not honor JUnit 5 display names
[ https://issues.apache.org/jira/browse/SUREFIRE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006314#comment-17006314 ] Geoff Soutter commented on SUREFIRE-1546: - Awesome work Tibor! I am particularly interested in the use case for DynamicTest and DynamicContainer, so I intend to test the new version and see how that works. I guess Stig will do / has done the same for ParameterizedTest. If I find it is not matching my expectations/hopes, next thing would be to create a pull request with an integration test which shows how I would like it to work, as you suggested. > JUnit 5 runner does not honor JUnit 5 display names > --- > > Key: SUREFIRE-1546 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1546 > Project: Maven Surefire > Issue Type: Bug > Components: JUnit 5.x support >Affects Versions: 2.22.0 >Reporter: Romain Manni-Bucau >Assignee: Tibor Digana >Priority: Major > Labels: junit5 > Fix For: 3.0.0-M4 > > Time Spent: 40m > Remaining Estimate: 0h > > JUnit 5 runner should respect the test @DisplayName instead of displaying the > classname if any is defined. Seems last release doesn't support that feature > of JUnit 5 making the console output and reports not the expected ones. > > Origin: https://github.com/junit-team/junit5/issues/990 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1546) JUnit 5 runner does not honor JUnit 5 display names
[
https://issues.apache.org/jira/browse/SUREFIRE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777610#comment-16777610
]
Geoff Soutter commented on SUREFIRE-1546:
-
I had a quick play with 3.0.0-M4-SNAPSHOT and a JUnit5 DynamicTest. I noticed
Filename is {{TEST-}}
Content of file has
{code}
JUnit 5 runner does not honor JUnit 5 display names
> ---
>
> Key: SUREFIRE-1546
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1546
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support
>Affects Versions: 2.22.0
>Reporter: Romain Manni-Bucau
>Assignee: Tibor Digana
>Priority: Major
> Labels: junit5
> Fix For: 3.0.0-M4
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> JUnit 5 runner should respect the test @DisplayName instead of displaying the
> classname if any is defined. Seems last release doesn't support that feature
> of JUnit 5 making the console output and reports not the expected ones.
>
> Origin: https://github.com/junit-team/junit5/issues/990
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
