Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
Hi devs,
I hope you have spent a nice weekend ;-)
Integrated new Surefire feature [1] in the core-its branch [2] which fixed
the first issue (see point 1.) and the build is successful [3].
After making an analysis, with a help of the logs, we can see that there
are 3 known issues at least:
1. Surefire forked JVM should not be spied by Agent of Jenkins. This Agent
is mandatory in Maven build process.
Using unnecessary agent in Surefire JVM would mean that the tests fail
with well known JUnit4 assertion failures [4].
Removing JAVA_TOOL_OPTIONS cut the agent off in the tests.
[proof by Stephen Connolly] Ensure any forked Maven executions are not
spied on by Jenkins
2. a JVM warning is printed in standard output and caused by the commit [5]
where the tests expected clean stream, see [6].
The build [7] contains the commit 690841e9 and fails. This was found and
proved by bisect of comits.
3. MAVEN_OPTS
There are several variants.
Herve's investigation which is fine, and
mine [8] which is an improvement taking developer's mavenOpts from his
build into Surefire's forked JVM.
Additionally, I skipped [9] global env.MAVEN_OPTS from '.mavenrc' in
order to make sure that the influence of the system (hardware) is avoided
and I can fully consider concentrate only on software issues.
So this helps me to separate the influence and avoid a mix of influences
in my investigation process.
[1]:
https://github.com/apache/maven-surefire/commit/065001b2e6417126fd80d5f1a5523a3469e3dd5a
[2]:
https://github.com/apache/maven-integration-testing/commit/5be07fd52727ef09dcc29539e80fb6aff534ad3b#diff-600376dffeb79835ede4a0b285078036R157
[3]:
https://builds.apache.org/job/maven-box/job/maven/job/removed-java-tools-2/2/
[4]: expected:<[]> but was:<[Picked up JAVA_TOOL_OPTIONS: ...]>
[5]: 7/24/2019, 690841e9, [MNG-6697]
https://github.com/apache/maven/pull/261#issuecomment-517999286
[6]: expected:<[]> but was:<[WARNING: An illegal reflective access
operation has occurred ... ]>.
For more information see [5].
[7]:
https://builds.apache.org/job/maven-box/job/maven/job/removed-java-tools-3/
[8]:
https://github.com/apache/maven-integration-testing/commit/5a40bb177b5ad3851cabf20c7983998d76258860#diff-089d3294860484e2addcd4d75bf8596fR352
https://github.com/apache/maven/commit/ad58ca489038829d5c9d54d7201ddf3eac49ad20
[9]:
https://github.com/apache/maven/commit/e0a67951a8c5258546c48cd45a74a31a71b74424
Cheers
Tibor17
On Sat, Aug 3, 2019 at 9:12 PM Robert Scholte wrote:
> Here's a very short summary and conclusion about what happened the last
> period.
>
> Half July some integration tests of Maven started to fail, but not always
> (it good happen that a certain JDK would succeed again with the next
> run).
> At that time we only noticed:
> - it failed only on the Linux nodes of Jenkins.
> - the failing tests were using MAVEN_OPTS
>
> It took quite a while, but in the end Hervé discovered the root cause:
> - On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
> that set the value of MAVEN_OPTS
>
> INFRA discovered that it was the incubator-shardingsphere that created
> this file when building there project.
>
> We've provided a PR for this project together with an explanation. This
> PR
> has been accepted and INFRA has removed the .mavenrc files.
>
> Over the past weeks a lot of changes has been done to the code for
> investigation.
> All changes that were done to change the behavior related to environments
> variables and mavenrc file should and will be reverted.
> Mavens master branch was stable, so to ensure the right things are still
> tested the need to be restored (some of them did indeed test the
> MAVEN_OPTS explicitly as issued in its JIRA).
> In general we don't change ITs, we only extend then and provide a range
> to
> control for which Maven versions the test is valid.
>
> Details and discussions can be found on the slack channel.
>
> thanks,
> Robert
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>
Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
Well, the Maven project is not the only one who wants to ensure repeatable
builds and may be surprises by things like that. It certainly would be a
safeguard for anybody interested in explizit configurations for CI jobs.
--
http://bernd.eckenfels.net
Von: Robert Scholte
Gesendet: Sonntag, August 4, 2019 8:26 PM
An: Maven Developers List
Betreff: Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
No, there's no need for that and it would over-complicate the scripts.
mavenrc can be handy, but only at INFRA level.
As said, we were able to fix it on the project-level. The project simply
wasn't aware of
- side effects and impact of creating this file on Jenkins
- the existence of .mvn/jvm.config
Robert
On Sun, 04 Aug 2019 18:51:52 +0200, Bernd Eckenfels
wrote:
> Do we need a Maven Option to turn of processing of implicit
> configurations, just like you can turn of shellrc Files?
>
>
> --
> http://bernd.eckenfels.net
>
>
> Von: Robert Scholte
> Gesendet: Samstag, August 3, 2019 9:12 PM
> An: Maven Developers List
> Betreff: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
>
> Here's a very short summary and conclusion about what happened the last
> period.
>
> Half July some integration tests of Maven started to fail, but not always
> (it good happen that a certain JDK would succeed again with the next
> run).
> At that time we only noticed:
> - it failed only on the Linux nodes of Jenkins.
> - the failing tests were using MAVEN_OPTS
>
> It took quite a while, but in the end Hervé discovered the root cause:
> - On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
> that set the value of MAVEN_OPTS
>
> INFRA discovered that it was the incubator-shardingsphere that created
> this file when building there project.
>
> We've provided a PR for this project together with an explanation. This
> PR
> has been accepted and INFRA has removed the .mavenrc files.
>
> Over the past weeks a lot of changes has been done to the code for
> investigation.
> All changes that were done to change the behavior related to environments
> variables and mavenrc file should and will be reverted.
> Mavens master branch was stable, so to ensure the right things are still
> tested the need to be restored (some of them did indeed test the
> MAVEN_OPTS explicitly as issued in its JIRA).
> In general we don't change ITs, we only extend then and provide a range
> to
> control for which Maven versions the test is valid.
>
> Details and discussions can be found on the slack channel.
>
> thanks,
> Robert
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
No, there's no need for that and it would over-complicate the scripts.
mavenrc can be handy, but only at INFRA level.
As said, we were able to fix it on the project-level. The project simply
wasn't aware of
- side effects and impact of creating this file on Jenkins
- the existence of .mvn/jvm.config
Robert
On Sun, 04 Aug 2019 18:51:52 +0200, Bernd Eckenfels
wrote:
Do we need a Maven Option to turn of processing of implicit
configurations, just like you can turn of shellrc Files?
--
http://bernd.eckenfels.net
Von: Robert Scholte
Gesendet: Samstag, August 3, 2019 9:12 PM
An: Maven Developers List
Betreff: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
Here's a very short summary and conclusion about what happened the last
period.
Half July some integration tests of Maven started to fail, but not always
(it good happen that a certain JDK would succeed again with the next
run).
At that time we only noticed:
- it failed only on the Linux nodes of Jenkins.
- the failing tests were using MAVEN_OPTS
It took quite a while, but in the end Hervé discovered the root cause:
- On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
that set the value of MAVEN_OPTS
INFRA discovered that it was the incubator-shardingsphere that created
this file when building there project.
We've provided a PR for this project together with an explanation. This
PR
has been accepted and INFRA has removed the .mavenrc files.
Over the past weeks a lot of changes has been done to the code for
investigation.
All changes that were done to change the behavior related to environments
variables and mavenrc file should and will be reverted.
Mavens master branch was stable, so to ensure the right things are still
tested the need to be restored (some of them did indeed test the
MAVEN_OPTS explicitly as issued in its JIRA).
In general we don't change ITs, we only extend then and provide a range
to
control for which Maven versions the test is valid.
Details and discussions can be found on the slack channel.
thanks,
Robert
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
Do we need a Maven Option to turn of processing of implicit configurations,
just like you can turn of shellrc Files?
--
http://bernd.eckenfels.net
Von: Robert Scholte
Gesendet: Samstag, August 3, 2019 9:12 PM
An: Maven Developers List
Betreff: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
Here's a very short summary and conclusion about what happened the last
period.
Half July some integration tests of Maven started to fail, but not always
(it good happen that a certain JDK would succeed again with the next run).
At that time we only noticed:
- it failed only on the Linux nodes of Jenkins.
- the failing tests were using MAVEN_OPTS
It took quite a while, but in the end Hervé discovered the root cause:
- On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
that set the value of MAVEN_OPTS
INFRA discovered that it was the incubator-shardingsphere that created
this file when building there project.
We've provided a PR for this project together with an explanation. This PR
has been accepted and INFRA has removed the .mavenrc files.
Over the past weeks a lot of changes has been done to the code for
investigation.
All changes that were done to change the behavior related to environments
variables and mavenrc file should and will be reverted.
Mavens master branch was stable, so to ensure the right things are still
tested the need to be restored (some of them did indeed test the
MAVEN_OPTS explicitly as issued in its JIRA).
In general we don't change ITs, we only extend then and provide a range to
control for which Maven versions the test is valid.
Details and discussions can be found on the slack channel.
thanks,
Robert
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
On Sun, 04 Aug 2019 13:06:31 +0200, Elliotte Rusty Harold
wrote:
This investigation and resolution of this mystery has been quite
informative. I suspect there are general principles and lessons here
about separating infrastructure for different projects that extend
well beyond Maven itself and probably beyond Jenkins. Containers might
play a role.
Using containers will isolate some issues, but I prefer to test Maven as
it is used by its users. Even though I don't have the numbers, I still
think most use it directly on the OS. The usage of Maven inside a
container is probably increasing, so it does make sense if such a setup is
possible with the infra of the ASF.
Perhaps someone would be interested in more formally publishing a full
writeup somewhere? Or perhaps delivering a lightning talk on this at
ApacheCon? This feels quite valuable for general DevOps.
This is indeed an interesting topic where one must understand how all the
tools work, especially in a shared environment.
AFAIK all Maven related proposals were rejected for ApacheCon Europe.
Robert
On Sat, Aug 3, 2019 at 3:12 PM Robert Scholte
wrote:
Here's a very short summary and conclusion about what happened the last
period.
Half July some integration tests of Maven started to fail, but not
always
(it good happen that a certain JDK would succeed again with the next
run).
At that time we only noticed:
- it failed only on the Linux nodes of Jenkins.
- the failing tests were using MAVEN_OPTS
It took quite a while, but in the end Hervé discovered the root cause:
- On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
that set the value of MAVEN_OPTS
INFRA discovered that it was the incubator-shardingsphere that created
this file when building there project.
We've provided a PR for this project together with an explanation. This
PR
has been accepted and INFRA has removed the .mavenrc files.
Over the past weeks a lot of changes has been done to the code for
investigation.
All changes that were done to change the behavior related to
environments
variables and mavenrc file should and will be reverted.
Mavens master branch was stable, so to ensure the right things are still
tested the need to be restored (some of them did indeed test the
MAVEN_OPTS explicitly as issued in its JIRA).
In general we don't change ITs, we only extend then and provide a range
to
control for which Maven versions the test is valid.
Details and discussions can be found on the slack channel.
thanks,
Robert
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
This investigation and resolution of this mystery has been quite
informative. I suspect there are general principles and lessons here
about separating infrastructure for different projects that extend
well beyond Maven itself and probably beyond Jenkins. Containers might
play a role.
Perhaps someone would be interested in more formally publishing a full
writeup somewhere? Or perhaps delivering a lightning talk on this at
ApacheCon? This feels quite valuable for general DevOps.
On Sat, Aug 3, 2019 at 3:12 PM Robert Scholte wrote:
>
> Here's a very short summary and conclusion about what happened the last
> period.
>
> Half July some integration tests of Maven started to fail, but not always
> (it good happen that a certain JDK would succeed again with the next run).
> At that time we only noticed:
> - it failed only on the Linux nodes of Jenkins.
> - the failing tests were using MAVEN_OPTS
>
> It took quite a while, but in the end Hervé discovered the root cause:
> - On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
> that set the value of MAVEN_OPTS
>
> INFRA discovered that it was the incubator-shardingsphere that created
> this file when building there project.
>
> We've provided a PR for this project together with an explanation. This PR
> has been accepted and INFRA has removed the .mavenrc files.
>
> Over the past weeks a lot of changes has been done to the code for
> investigation.
> All changes that were done to change the behavior related to environments
> variables and mavenrc file should and will be reverted.
> Mavens master branch was stable, so to ensure the right things are still
> tested the need to be restored (some of them did indeed test the
> MAVEN_OPTS explicitly as issued in its JIRA).
> In general we don't change ITs, we only extend then and provide a range to
> control for which Maven versions the test is valid.
>
> Details and discussions can be found on the slack channel.
>
> thanks,
> Robert
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
--
Elliotte Rusty Harold
elh...@ibiblio.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
