subject:"Re\: 3 ITs permanently fail on Linux \/ Maven Core"

Re: 3 ITs permanently fail on Linux / Maven Core

2019-08-02 Thread Hervé BOUTEMY

here it is: issue fixed

as you can see by reading the INFRA-18812 Jira issue [1]:
- .mavenrc file polluting Jenkins slaves was created by Shardingshpere project
- Robert proposed a PR to the project
- INFRA removed unwanted .mavenrc files

now the ITs depending on MAVEN_OPTS are back working
https://builds.apache.org/job/maven-box/job/maven/job/MAVEN_OPTS/

time to fix other issues (and drop workarounds prepared while investigating)

Regards,

Hervé

[1] https://issues.apache.org/jira/browse/INFRA-18812

Le mardi 30 juillet 2019, 07:57:27 CEST Hervé BOUTEMY a écrit :
> INFRA-18812 created
> 
> https://issues.apache.org/jira/browse/INFRA-18812
> 
> Le mardi 30 juillet 2019, 07:39:49 CEST Hervé BOUTEMY a écrit :
> > I found the root cause of failing IT on some Linux nodes!!!
> > 
> > 
> > activating debug mode for mvn shell script, we get:
> > + [ -f /home/jenkins/.mavenrc ]
> > + . /home/jenkins/.mavenrc
> > + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
> > 
> > there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> > instead of appending: I did not yet report to INFRA, need to check if this
> > variable is defined with Puppet and provide a PR
> > 
> > FYI, core ITs log files have finally always been accessible: just need to
> > look inside
> > org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT
> > -
> > tests.jar artifact
> > 
> > 
> > This debugging session has been hard, but I'm happy to say that it is now
> > under control: no more wild guesses about strange root causes, but pure
> > simple logic.
> > 
> > Regards,
> > 
> > Hervé
> > 
> > Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > > We discuss this issue in Slack
> > > We know for certain that there is a problem with environment variables.
> > > So there must be a bug in our code then. I have analysed the code I
> > > think
> > > the problem should be with spaces in the value of environment variable
> > > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > > when we have such rough element in String[] when we call
> > > Runtime.execute(cli, String[]).
> > > 
> > > Here is the list of comments from Slack and the way how we got there:
> > > 
> > > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > > the IT but I got to `execute()` method and I was interested in `String[]
> > > environment = getEnvironmentVariables();` which finally is passed to
> > > Java
> > > 
> > > running forked process. Please tell me if this code is okay for you:
> > > ```public String[] getEnvironmentVariables()
> > > {
> > > 
> > > addSystemEnvironment();
> > > String[] environmentVars = new String[envVars.size()];
> > > int i = 0;
> > > for ( String name : envVars.keySet() )
> > > {
> > > 
> > > String value = envVars.get( name );
> > > environmentVars[i] = name + "=" + value;
> > > i++;
> > > 
> > > }
> > > return environmentVars;
> > > 
> > > }```
> > > 
> > > I do not see quotation marks. This is important if you use two or more
> > > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > > Next issue is that the env vars of the parent process are merged with
> > > given
> > > vars for the child process in `addSystemEnvironment();`, Line 239.
> > > 
> > > that's why maybe platform Linux does not understand the space and maybe
> > > Windows does not have a problem with it.
> > > 
> > > merging env vars is strange because of JAVA_HOME, there are at least
> > > two.
> > > One in CLI and second in env vars.
> > > 
> > > I think we should use latest `maven-shared-utils` and check the code in
> > > shared utils. Maybe to fix shared utils and meanwhile use snapshot
> > > version,
> > > maybe.
> > > 
> > > 
> > > 
> > > 
> > > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana 
> > > 
> > > wrote:
> > > > Heads up!,
> > > > 
> > > > I am investigating the build errors on Maven Core.
> > > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > > --hard HEAD~12) and observed the outcome.
> > > > 
> > > > I have investigated 29 commits. Not sure how far to go...
> > > > 
> > > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8,
> > > > 11,
> > > > 12
> > > > (16 ITs)
> > > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8
> > > > (8
> > > > ITs)
> > > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > > > 
> > > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > > Always related to Linux.
> > > > 
> > > > See the list of errors and branches:
> > > > 
> > > > 
> > > > https://builds.apache.org/job/maven-box/job/maven/j

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-30 Thread Tibor Digana

ok Herve, thank You very much. I am watching the build . Perhaps we should
fix our code first and then report an issue to INFRA because maybe they
have a reason why they are providing their setup on their machines.

On Tue, Jul 30, 2019 at 2:01 PM Hervé BOUTEMY  wrote:

> look at debug trace:
> [ -f /home/jenkins/.mavenrc ]
> + . /home/jenkins/.mavenrc
> + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
>
> there is no "global" and "local" env variables: there is a script (/home/
> jenkins/.mavenrc) that defines MAVEN_OPTS env variable, then overrides
> previous
> value that was set in our IT code
>
> such script should define:
> MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m ${MAVEN_OPTS}
>
>
> this will solve issues of Maven core ITs requiring MAVEN_OPTS
>
> I don't know for maven-archetype, I'm not working on it, and this thread
> is
> not about it: I don't know why you're mixing everything and make simple
> things
> complex
>
> Regards,
>
> Hervé
>
> Le mardi 30 juillet 2019, 09:40:43 CEST Tibor Digana a écrit :
> > Essentially you want to say that global environment variable overrides
> > local MAVEN_OPTS?
> > This way I have undertood your email.
> > but I would say that it cannot be!
> >
> > Download the Maven dist and open the scripts $MVN_HOME/bin/mvn and there
> we
> > will see this.
> > This script in our distcorresponds to what I see in your email.
> > So I think all we have to do is to export MAVEN_SKIP_RC to 1 in our ITs
> and
> > prevent from merging global and local environment variables.
> >
> > if [ -z "$MAVEN_SKIP_RC" ] ; then
> >   if [ -f /etc/mavenrc ] ; then
> > . /etc/mavenrc
> >   fi
> >   if [ -f "$HOME/.mavenrc" ] ; then
> > . "$HOME/.mavenrc"
> >   fi
> > fi
> > ...
> > MAVEN_OPTS="`concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config"`
> > $MAVEN_OPTS"
> >
> > We have a lots of troubles with Jenkins.
> > Solving this does not mean anything becase we ca see in the logs that
> > "apache/maven-archetype" randomly lost files.
> > We have problem with networ connectivity to Maven Central, every second
> day
> > and more ofter in the night of European time.
> >
> > Cheers
> > Tibor17
> >
> > On Tue, Jul 30, 2019 at 7:40 AM Hervé BOUTEMY 
> wrote:
> > > I found the root cause of failing IT on some Linux nodes!!!
> > >
> > >
> > > activating debug mode for mvn shell script, we get:
> > > + [ -f /home/jenkins/.mavenrc ]
> > > + . /home/jenkins/.mavenrc
> > > + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
> > >
> > > there is a .mavenrc script on some Linux nodes that overrides
> MAVEN_OPTS
> > > instead of appending: I did not yet report to INFRA, need to check if
> this
> > > variable is defined with Puppet and provide a PR
> > >
> > > FYI, core ITs log files have finally always been accessible: just need
> to
> > > look inside
> > >
> org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT
> > > -tests.jar artifact
> > >
> > >
> > > This debugging session has been hard, but I'm happy to say that it is
> now
> > > under control: no more wild guesses about strange root causes, but pure
> > > simple logic.
> > >
> > > Regards,
> > >
> > > Hervé
> > >
> > > Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > > > We discuss this issue in Slack
> > > > We know for certain that there is a problem with environment
> variables.
> > > > So there must be a bug in our code then. I have analysed the code I
> > > > think
> > > > the problem should be with spaces in the value of environment
> variable
> > > > MAVEN_OPTS. There is no quotation marks around such environemnt
> variable
> > > > when we have such rough element in String[] when we call
> > > > Runtime.execute(cli, String[]).
> > > >
> > > > Here is the list of comments from Slack and the way how we got there:
> > > >
> > > > the `forkMode=auto` in profile `embedded` so it's okay. But I
> analyzed
> > > > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > > > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > > > `maven-shared-utils:0.9`. Although forking is used if we use env
> vars in
> > > > the IT but I got to `execute()` method and I was interested in
> `String[]
> > > > environment = getEnvironmentVariables();` which finally is passed to
> > > > Java
> > > >
> > > > running forked process. Please tell me if this code is okay for you:
> > > > ```public String[] getEnvironmentVariables()
> > > > {
> > > >
> > > > addSystemEnvironment();
> > > > String[] environmentVars = new String[envVars.size()];
> > > > int i = 0;
> > > > for ( String name : envVars.keySet() )
> > > > {
> > > >
> > > > String value = envVars.get( name );
> > > > environmentVars[i] = name + "=" + value;
> > > > i++;
> > > >
> > > > }
> > > > return environmentVars;
> > > >
> > > > }```
> > > >
> > > > I do not see quotation marks. This is important if you use two or
> more
> > > > systm propert

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-30 Thread Hervé BOUTEMY

look at debug trace:
[ -f /home/jenkins/.mavenrc ]
+ . /home/jenkins/.mavenrc
+ MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m

there is no "global" and "local" env variables: there is a script (/home/
jenkins/.mavenrc) that defines MAVEN_OPTS env variable, then overrides previous 
value that was set in our IT code

such script should define:
MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m ${MAVEN_OPTS}


this will solve issues of Maven core ITs requiring MAVEN_OPTS

I don't know for maven-archetype, I'm not working on it, and this thread is 
not about it: I don't know why you're mixing everything and make simple things 
complex

Regards,

Hervé

Le mardi 30 juillet 2019, 09:40:43 CEST Tibor Digana a écrit :
> Essentially you want to say that global environment variable overrides
> local MAVEN_OPTS?
> This way I have undertood your email.
> but I would say that it cannot be!
> 
> Download the Maven dist and open the scripts $MVN_HOME/bin/mvn and there we
> will see this.
> This script in our distcorresponds to what I see in your email.
> So I think all we have to do is to export MAVEN_SKIP_RC to 1 in our ITs and
> prevent from merging global and local environment variables.
> 
> if [ -z "$MAVEN_SKIP_RC" ] ; then
>   if [ -f /etc/mavenrc ] ; then
> . /etc/mavenrc
>   fi
>   if [ -f "$HOME/.mavenrc" ] ; then
> . "$HOME/.mavenrc"
>   fi
> fi
> ...
> MAVEN_OPTS="`concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config"`
> $MAVEN_OPTS"
> 
> We have a lots of troubles with Jenkins.
> Solving this does not mean anything becase we ca see in the logs that
> "apache/maven-archetype" randomly lost files.
> We have problem with networ connectivity to Maven Central, every second day
> and more ofter in the night of European time.
> 
> Cheers
> Tibor17
> 
> On Tue, Jul 30, 2019 at 7:40 AM Hervé BOUTEMY  wrote:
> > I found the root cause of failing IT on some Linux nodes!!!
> > 
> > 
> > activating debug mode for mvn shell script, we get:
> > + [ -f /home/jenkins/.mavenrc ]
> > + . /home/jenkins/.mavenrc
> > + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
> > 
> > there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> > instead of appending: I did not yet report to INFRA, need to check if this
> > variable is defined with Puppet and provide a PR
> > 
> > FYI, core ITs log files have finally always been accessible: just need to
> > look inside
> > org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT
> > -tests.jar artifact
> > 
> > 
> > This debugging session has been hard, but I'm happy to say that it is now
> > under control: no more wild guesses about strange root causes, but pure
> > simple logic.
> > 
> > Regards,
> > 
> > Hervé
> > 
> > Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > > We discuss this issue in Slack
> > > We know for certain that there is a problem with environment variables.
> > > So there must be a bug in our code then. I have analysed the code I
> > > think
> > > the problem should be with spaces in the value of environment variable
> > > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > > when we have such rough element in String[] when we call
> > > Runtime.execute(cli, String[]).
> > > 
> > > Here is the list of comments from Slack and the way how we got there:
> > > 
> > > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > > the IT but I got to `execute()` method and I was interested in `String[]
> > > environment = getEnvironmentVariables();` which finally is passed to
> > > Java
> > > 
> > > running forked process. Please tell me if this code is okay for you:
> > > ```public String[] getEnvironmentVariables()
> > > {
> > > 
> > > addSystemEnvironment();
> > > String[] environmentVars = new String[envVars.size()];
> > > int i = 0;
> > > for ( String name : envVars.keySet() )
> > > {
> > > 
> > > String value = envVars.get( name );
> > > environmentVars[i] = name + "=" + value;
> > > i++;
> > > 
> > > }
> > > return environmentVars;
> > > 
> > > }```
> > > 
> > > I do not see quotation marks. This is important if you use two or more
> > > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > > Next issue is that the env vars of the parent process are merged with
> > 
> > given
> > 
> > > vars for the child process in `addSystemEnvironment();`, Line 239.
> > > 
> > > that's why maybe platform Linux does not understand the space and maybe
> > > Windows does not have a problem with it.
> > > 
> > > merging env vars is strange because of JAVA_HOME, there are at least
> > > two.
> > > One in CLI and second in env vars.
> > > 
>

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-30 Thread Tibor Digana

Jenkins build is in progress and waiting for next available executor.
I think we will have the result today evening.
Shortly I skipped '.mavenrc' merging, see [1] and I expect successful
ITs(0553, 4747, 4590) and I expect newly failed IT (6720) which should be
orthogonal to our current problem.
For more information see the Jira [2].
[1]: https://builds.apache.org/job/maven-box/job/maven/job/master/
[2]: https://issues.apache.org/jira/browse/MNG-6725

On Tue, Jul 30, 2019 at 9:40 AM Tibor Digana  wrote:

> Essentially you want to say that global environment variable overrides
> local MAVEN_OPTS?
> This way I have undertood your email.
> but I would say that it cannot be!
>
> Download the Maven dist and open the scripts $MVN_HOME/bin/mvn and there
> we will see this.
> This script in our distcorresponds to what I see in your email.
> So I think all we have to do is to export MAVEN_SKIP_RC to 1 in our ITs
> and prevent from merging global and local environment variables.
>
> if [ -z "$MAVEN_SKIP_RC" ] ; then
>   if [ -f /etc/mavenrc ] ; then
> . /etc/mavenrc
>   fi
>   if [ -f "$HOME/.mavenrc" ] ; then
> . "$HOME/.mavenrc"
>   fi
> fi
> ...
> MAVEN_OPTS="`concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config"`
> $MAVEN_OPTS"
>
> We have a lots of troubles with Jenkins.
> Solving this does not mean anything becase we ca see in the logs that
> "apache/maven-archetype" randomly lost files.
> We have problem with networ connectivity to Maven Central, every second
> day and more ofter in the night of European time.
>
> Cheers
> Tibor17
>
>
>
> On Tue, Jul 30, 2019 at 7:40 AM Hervé BOUTEMY 
> wrote:
>
>> I found the root cause of failing IT on some Linux nodes!!!
>>
>>
>> activating debug mode for mvn shell script, we get:
>> + [ -f /home/jenkins/.mavenrc ]
>> + . /home/jenkins/.mavenrc
>> + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
>>
>> there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
>> instead of appending: I did not yet report to INFRA, need to check if this
>> variable is defined with Puppet and provide a PR
>>
>> FYI, core ITs log files have finally always been accessible: just need to
>> look inside
>> org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT-tests.jar
>> artifact
>>
>>
>> This debugging session has been hard, but I'm happy to say that it is now
>> under control: no more wild guesses about strange root causes, but pure
>> simple logic.
>>
>> Regards,
>>
>> Hervé
>>
>> Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
>> > We discuss this issue in Slack
>> > We know for certain that there is a problem with environment variables.
>> > So there must be a bug in our code then. I have analysed the code I
>> think
>> > the problem should be with spaces in the value of environment variable
>> > MAVEN_OPTS. There is no quotation marks around such environemnt variable
>> > when we have such rough element in String[] when we call
>> > Runtime.execute(cli, String[]).
>> >
>> > Here is the list of comments from Slack and the way how we got there:
>> >
>> > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
>> > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
>> > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
>> > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
>> > the IT but I got to `execute()` method and I was interested in `String[]
>> > environment = getEnvironmentVariables();` which finally is passed to
>> Java
>> > running forked process. Please tell me if this code is okay for you:
>> > ```public String[] getEnvironmentVariables()
>> > {
>> > addSystemEnvironment();
>> > String[] environmentVars = new String[envVars.size()];
>> > int i = 0;
>> > for ( String name : envVars.keySet() )
>> > {
>> > String value = envVars.get( name );
>> > environmentVars[i] = name + "=" + value;
>> > i++;
>> > }
>> > return environmentVars;
>> > }```
>> > I do not see quotation marks. This is important if you use two or more
>> > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
>> > Next issue is that the env vars of the parent process are merged with
>> given
>> > vars for the child process in `addSystemEnvironment();`, Line 239.
>> >
>> > that's why maybe platform Linux does not understand the space and maybe
>> > Windows does not have a problem with it.
>> >
>> > merging env vars is strange because of JAVA_HOME, there are at least
>> two.
>> > One in CLI and second in env vars.
>> >
>> > I think we should use latest `maven-shared-utils` and check the code in
>> > shared utils. Maybe to fix shared utils and meanwhile use snapshot
>> version,
>> > maybe.
>> >
>> >
>> >
>> >
>> > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana 
>> >
>> > wrote:
>> > > Heads up!,
>> > >
>> > > I am investigating the build errors on Maven Core.
>> > > So I created t

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-30 Thread Tibor Digana

Essentially you want to say that global environment variable overrides
local MAVEN_OPTS?
This way I have undertood your email.
but I would say that it cannot be!

Download the Maven dist and open the scripts $MVN_HOME/bin/mvn and there we
will see this.
This script in our distcorresponds to what I see in your email.
So I think all we have to do is to export MAVEN_SKIP_RC to 1 in our ITs and
prevent from merging global and local environment variables.

if [ -z "$MAVEN_SKIP_RC" ] ; then
  if [ -f /etc/mavenrc ] ; then
. /etc/mavenrc
  fi
  if [ -f "$HOME/.mavenrc" ] ; then
. "$HOME/.mavenrc"
  fi
fi
...
MAVEN_OPTS="`concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config"`
$MAVEN_OPTS"

We have a lots of troubles with Jenkins.
Solving this does not mean anything becase we ca see in the logs that
"apache/maven-archetype" randomly lost files.
We have problem with networ connectivity to Maven Central, every second day
and more ofter in the night of European time.

Cheers
Tibor17



On Tue, Jul 30, 2019 at 7:40 AM Hervé BOUTEMY  wrote:

> I found the root cause of failing IT on some Linux nodes!!!
>
>
> activating debug mode for mvn shell script, we get:
> + [ -f /home/jenkins/.mavenrc ]
> + . /home/jenkins/.mavenrc
> + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
>
> there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> instead of appending: I did not yet report to INFRA, need to check if this
> variable is defined with Puppet and provide a PR
>
> FYI, core ITs log files have finally always been accessible: just need to
> look inside
> org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT-tests.jar
> artifact
>
>
> This debugging session has been hard, but I'm happy to say that it is now
> under control: no more wild guesses about strange root causes, but pure
> simple logic.
>
> Regards,
>
> Hervé
>
> Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > We discuss this issue in Slack
> > We know for certain that there is a problem with environment variables.
> > So there must be a bug in our code then. I have analysed the code I think
> > the problem should be with spaces in the value of environment variable
> > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > when we have such rough element in String[] when we call
> > Runtime.execute(cli, String[]).
> >
> > Here is the list of comments from Slack and the way how we got there:
> >
> > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > the IT but I got to `execute()` method and I was interested in `String[]
> > environment = getEnvironmentVariables();` which finally is passed to Java
> > running forked process. Please tell me if this code is okay for you:
> > ```public String[] getEnvironmentVariables()
> > {
> > addSystemEnvironment();
> > String[] environmentVars = new String[envVars.size()];
> > int i = 0;
> > for ( String name : envVars.keySet() )
> > {
> > String value = envVars.get( name );
> > environmentVars[i] = name + "=" + value;
> > i++;
> > }
> > return environmentVars;
> > }```
> > I do not see quotation marks. This is important if you use two or more
> > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > Next issue is that the env vars of the parent process are merged with
> given
> > vars for the child process in `addSystemEnvironment();`, Line 239.
> >
> > that's why maybe platform Linux does not understand the space and maybe
> > Windows does not have a problem with it.
> >
> > merging env vars is strange because of JAVA_HOME, there are at least two.
> > One in CLI and second in env vars.
> >
> > I think we should use latest `maven-shared-utils` and check the code in
> > shared utils. Maybe to fix shared utils and meanwhile use snapshot
> version,
> > maybe.
> >
> >
> >
> >
> > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana 
> >
> > wrote:
> > > Heads up!,
> > >
> > > I am investigating the build errors on Maven Core.
> > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > --hard HEAD~12) and observed the outcome.
> > >
> > > I have investigated 29 commits. Not sure how far to go...
> > >
> > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8,
> 11, 12
> > > (16 ITs)
> > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8
> (8
> > > ITs)
> > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > >
> > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > Always related to Linux.
> > >
> > > See the list of errors and branches:
> > >
> > >
> > >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-r

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-29 Thread Hervé BOUTEMY

INFRA-18812 created

https://issues.apache.org/jira/browse/INFRA-18812

Le mardi 30 juillet 2019, 07:39:49 CEST Hervé BOUTEMY a écrit :
> I found the root cause of failing IT on some Linux nodes!!!
> 
> 
> activating debug mode for mvn shell script, we get:
> + [ -f /home/jenkins/.mavenrc ]
> + . /home/jenkins/.mavenrc
> + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
> 
> there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> instead of appending: I did not yet report to INFRA, need to check if this
> variable is defined with Puppet and provide a PR
> 
> FYI, core ITs log files have finally always been accessible: just need to
> look inside
> org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT-
> tests.jar artifact
> 
> 
> This debugging session has been hard, but I'm happy to say that it is now
> under control: no more wild guesses about strange root causes, but pure
> simple logic.
> 
> Regards,
> 
> Hervé
> 
> Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > We discuss this issue in Slack
> > We know for certain that there is a problem with environment variables.
> > So there must be a bug in our code then. I have analysed the code I think
> > the problem should be with spaces in the value of environment variable
> > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > when we have such rough element in String[] when we call
> > Runtime.execute(cli, String[]).
> > 
> > Here is the list of comments from Slack and the way how we got there:
> > 
> > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > the IT but I got to `execute()` method and I was interested in `String[]
> > environment = getEnvironmentVariables();` which finally is passed to Java
> > 
> > running forked process. Please tell me if this code is okay for you:
> > ```public String[] getEnvironmentVariables()
> > {
> > 
> > addSystemEnvironment();
> > String[] environmentVars = new String[envVars.size()];
> > int i = 0;
> > for ( String name : envVars.keySet() )
> > {
> > 
> > String value = envVars.get( name );
> > environmentVars[i] = name + "=" + value;
> > i++;
> > 
> > }
> > return environmentVars;
> > 
> > }```
> > 
> > I do not see quotation marks. This is important if you use two or more
> > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > Next issue is that the env vars of the parent process are merged with
> > given
> > vars for the child process in `addSystemEnvironment();`, Line 239.
> > 
> > that's why maybe platform Linux does not understand the space and maybe
> > Windows does not have a problem with it.
> > 
> > merging env vars is strange because of JAVA_HOME, there are at least two.
> > One in CLI and second in env vars.
> > 
> > I think we should use latest `maven-shared-utils` and check the code in
> > shared utils. Maybe to fix shared utils and meanwhile use snapshot
> > version,
> > maybe.
> > 
> > 
> > 
> > 
> > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana 
> > 
> > wrote:
> > > Heads up!,
> > > 
> > > I am investigating the build errors on Maven Core.
> > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > --hard HEAD~12) and observed the outcome.
> > > 
> > > I have investigated 29 commits. Not sure how far to go...
> > > 
> > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11,
> > > 12
> > > (16 ITs)
> > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> > > ITs)
> > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > > 
> > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > Always related to Linux.
> > > 
> > > See the list of errors and branches:
> > > 
> > > 
> > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3
> > > .3
> > > -reset-head-12/1/#showFailuresLink
> > > 
> > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3
> > > .3
> > > -reset-head-14/1/
> > > 
> > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3
> > > .3
> > > -reset-head-29/1/
> > > 
> > > 
> > > 
> > > From the MavenITmng0553SettingsAuthzEncryptionTest:
> > > 
> > > [ERROR] Error executing Maven.
> > > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> > > java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml
> > > (No
> > > such file or directory)
> > > 
> > > at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > > 
> > > (SecUtil.java:69)
> > > 
> > > at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > > at org.apache

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-29 Thread Hervé BOUTEMY

I found the root cause of failing IT on some Linux nodes!!!


activating debug mode for mvn shell script, we get:
+ [ -f /home/jenkins/.mavenrc ]
+ . /home/jenkins/.mavenrc
+ MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m

there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS 
instead of appending: I did not yet report to INFRA, need to check if this 
variable is defined with Puppet and provide a PR

FYI, core ITs log files have finally always been accessible: just need to look 
inside 
org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT-tests.jar
 artifact


This debugging session has been hard, but I'm happy to say that it is now under 
control: no more wild guesses about strange root causes, but pure simple logic.

Regards,

Hervé

Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> We discuss this issue in Slack
> We know for certain that there is a problem with environment variables.
> So there must be a bug in our code then. I have analysed the code I think
> the problem should be with spaces in the value of environment variable
> MAVEN_OPTS. There is no quotation marks around such environemnt variable
> when we have such rough element in String[] when we call
> Runtime.execute(cli, String[]).
> 
> Here is the list of comments from Slack and the way how we got there:
> 
> the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> the IT but I got to `execute()` method and I was interested in `String[]
> environment = getEnvironmentVariables();` which finally is passed to Java
> running forked process. Please tell me if this code is okay for you:
> ```public String[] getEnvironmentVariables()
> {
> addSystemEnvironment();
> String[] environmentVars = new String[envVars.size()];
> int i = 0;
> for ( String name : envVars.keySet() )
> {
> String value = envVars.get( name );
> environmentVars[i] = name + "=" + value;
> i++;
> }
> return environmentVars;
> }```
> I do not see quotation marks. This is important if you use two or more
> systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> Next issue is that the env vars of the parent process are merged with given
> vars for the child process in `addSystemEnvironment();`, Line 239.
> 
> that's why maybe platform Linux does not understand the space and maybe
> Windows does not have a problem with it.
> 
> merging env vars is strange because of JAVA_HOME, there are at least two.
> One in CLI and second in env vars.
> 
> I think we should use latest `maven-shared-utils` and check the code in
> shared utils. Maybe to fix shared utils and meanwhile use snapshot version,
> maybe.
> 
> 
> 
> 
> On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana 
> 
> wrote:
> > Heads up!,
> > 
> > I am investigating the build errors on Maven Core.
> > So I created three branches moved the HEAD backwards (i.e. git reset
> > --hard HEAD~12) and observed the outcome.
> > 
> > I have investigated 29 commits. Not sure how far to go...
> > 
> > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11, 12
> > (16 ITs)
> > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> > ITs)
> > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > 
> > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > Always related to Linux.
> > 
> > See the list of errors and branches:
> > 
> > 
> > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> > -reset-head-12/1/#showFailuresLink
> > 
> > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> > -reset-head-14/1/
> > 
> > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> > -reset-head-29/1/
> > 
> > 
> > 
> > From the MavenITmng0553SettingsAuthzEncryptionTest:
> > 
> > [ERROR] Error executing Maven.
> > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> > java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml (No
> > such file or directory)
> > 
> > at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > 
> > (SecUtil.java:69)
> > 
> > at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> > at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> > 
> > [ERROR] Failed to execute goal on project test: Could not resolve
> > dependencies for project
> > org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
> > dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Failed
> > to read artifact descriptor for
> > org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not tran

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-28 Thread Tibor Digana

We discuss this issue in Slack
We know for certain that there is a problem with environment variables.
So there must be a bug in our code then. I have analysed the code I think
the problem should be with spaces in the value of environment variable
MAVEN_OPTS. There is no quotation marks around such environemnt variable
when we have such rough element in String[] when we call
Runtime.execute(cli, String[]).

Here is the list of comments from Slack and the way how we got there:

the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
`MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
testDir.getAbsolutePath() )` and I got to `CommandLine` class from
`maven-shared-utils:0.9`. Although forking is used if we use env vars in
the IT but I got to `execute()` method and I was interested in `String[]
environment = getEnvironmentVariables();` which finally is passed to Java
running forked process. Please tell me if this code is okay for you:
```public String[] getEnvironmentVariables()
{
addSystemEnvironment();
String[] environmentVars = new String[envVars.size()];
int i = 0;
for ( String name : envVars.keySet() )
{
String value = envVars.get( name );
environmentVars[i] = name + "=" + value;
i++;
}
return environmentVars;
}```
I do not see quotation marks. This is important if you use two or more
systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
Next issue is that the env vars of the parent process are merged with given
vars for the child process in `addSystemEnvironment();`, Line 239.

that's why maybe platform Linux does not understand the space and maybe
Windows does not have a problem with it.

merging env vars is strange because of JAVA_HOME, there are at least two.
One in CLI and second in env vars.

I think we should use latest `maven-shared-utils` and check the code in
shared utils. Maybe to fix shared utils and meanwhile use snapshot version,
maybe.

On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana 
wrote:

> Heads up!,
>
> I am investigating the build errors on Maven Core.
> So I created three branches moved the HEAD backwards (i.e. git reset
> --hard HEAD~12) and observed the outcome.
>
> I have investigated 29 commits. Not sure how far to go...
>
> `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11, 12
> (16 ITs)
> `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> ITs)
> `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
>
> Always the ITs 0553, 4590, 4747 fail on several nodes.
> Always related to Linux.
>
> See the list of errors and branches:
>
>
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-reset-head-12/1/#showFailuresLink
>
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-reset-head-14/1/
>
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-reset-head-29/1/
>
>
>
> From the MavenITmng0553SettingsAuthzEncryptionTest:
>
> [ERROR] Error executing Maven.
> org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml (No
> such file or directory)
> at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> (SecUtil.java:69)
> at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
>
> [ERROR] Failed to execute goal on project test: Could not resolve 
> dependencies for project org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: 
> Failed to collect dependencies at 
> org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Failed to read artifact 
> descriptor for org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not 
> transfer artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to 
> test (http://localhost:32917/): Not authorized
>
>
>
> From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:
>
> expected:
>  but 
> was:
>
>
>
> From the MavenITmng4747JavaAgentUsedByPluginTest:
>
> junit.framework.AssertionFailedError
>   at 
> org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit(MavenITmng4747JavaAgentUsedByPluginTest.java:63)
>
>

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-26 Thread Michael Osipov


Am 2019-07-20 um 12:25 schrieb Hervé BOUTEMY:

As a side note, I started by doing a lot of cleanup on old merged branches: it
would be nice if everybody did its own cleanup when merging


I have purged all of my merged branches, there are still a lot of open 
branches...



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-21 Thread Hervé BOUTEMY

I took time to launch the build many times and got to following conclusion:

- the only issue is about MAVEN_OPTS not being honoured in forked JVM
- it's not related to a JDK version but mode Jenkins node:
-> failing nodes: H24, H27, H28, H29, H33, H37
-> working nodes: H31, H32, H34, H44


to me, Maven core master branch is currently stable: the 4 ITs failing we see 
are caused by ASF Jenkins issues
If someone wants to launch a release, I'm fully supportive

Of course, if anybody has an idea on the root cause to the MAVEN_OPTS issue, 
I'm all ears open...

Regards,

Hervé

Le samedi 20 juillet 2019, 19:53:05 CEST Tibor Digana a écrit :
> Hello Herve,
> 
> no problem. I am investigating in
> https://builds.apache.org/job/maven-box/job/maven/job/EOL/
> the build is named after end-of-line because I have expected isues like
> that. Only Jenkins Linux build fail, pls my last two commits in the branch
> EOL.
> 
> My commits correspond to ".gitattributes" file and Jenkinsfile.
> In ".gitattributes" I set eol=LF and I am archiving the folder
> "core-it-suite" in Jenkinsfile.
> 
> It is strange that only Linux builds fail and thus there's is a kind of
> platform dependency as it seems in this error.
> I hope the archived ZIP file tells us more, e.g. logs at least.
> 
> 
> Definitely something is wrong on Jenkins/Linux or in Jenkinsfile because
> the build #22 was successful in
> https://builds.apache.org/job/maven-box/job/maven/job/MNG-6294/ one month
> ago, but I manually triggered #23 and I have got the same set of failed ITs
> as we have in master today.
> 
> Cheers
> Tibor
> 
> On Sat, Jul 20, 2019 at 7:25 PM Hervé BOUTEMY  wrote:
> > I don't get where you want to go: there is no disagreement on reverting
> > resolver from 1.4.0 to 1.3.3
> > 
> > the question is, once this revert has been done, about the 4 ITs failing
> > [2]:
> > master #242 linux-jdk7 / Run ITs Linux Java 7
> > 1. org.apache.maven.it
> > .MavenITmng0553SettingsAuthzEncryptionTest.testitEncryption
> > 2. org.apache.maven.it
> > .MavenITmng0553SettingsAuthzEncryptionTest.testitBasic
> > 3. org.apache.maven.it
> > .MavenITmng4590ImportedPomUsesSystemPropertiesTest.testit
> > 4. org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit
> > 
> > I investigated these ITs and found one common fact: they try to define
> > MAVEN_OPTS (and are the only ones during that)
> > 
> > verifier.setEnvironmentVariable( "MAVEN_OPTS", "" );
> > 
> > and this does not seem to be transferred to the verifier execution
> > 
> > I don't know yet why this does not work on some Jenkins nodes, trying to
> > add a few additional output to see if I can get some hints...
> > 
> > any idea welcome.
> > 
> > Regards,
> > 
> > Hervé
> > 
> > [2]
> > https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/
> > master/242/testReport/> 
> > Le samedi 20 juillet 2019, 12:48:27 CEST Tibor Digana a écrit :
> > > I wrote in our chat that Maven Core fails randomly on random JDK
> > > versions
> > > due to maven-resolver:1.4.0. It was proved by:
> > > 
> > > + Jenkins on Maven Core
> > > + Jira MNG-6714
> > > + logical analysis of code in resolver.
> > > 
> > > I wrote to Harald Wellmann that we should make code review together.
> > > In my profession and company, the multithreading was my Java Advanced
> > > expertize, so I am able to evaluate code which is thread safe and which
> > 
> > is
> > 
> > > not. Additionally, resolver:1.4.0 has logical problem even in single
> > 
> > thread
> > 
> > > which was proved by the branch where 11 unit tests failed on single
> > 
> > thread
> > 
> > > which of course must never happen.
> > > 
> > > These things prove me to say that MRESOLVER-7 is not applicable in
> > > production and we have to make code review together!
> > > 
> > > Sorry to say, I can teach few devs with JSR-133 and thread safety
> > 
> > regarding
> > 
> > > Java Memory Model, and then the devs would probably understand why I
> > 
> > found
> > 
> > > issues in the code. Nevertheless still there are two problems with the
> > > algorithm itself for whatever number of threads are executed:
> > > 
> > > + one object is concurrently modified across multiple threads and the
> > > object unexpectedly changes the state of the object
> > > + there is one loop which resolves version but the last iteration wins,
> > 
> > so
> > 
> > > it loop looks to me ilogical and causes unnecessary object state
> > > modifications
> > > 
> > > 
> > > To be unbiased, all these things have to be clarified in PR on GitHub,
> > > dicsussed, and changes have to be made in order to provide STABLE
> > 
> > resolver.
> > 
> > > That's all I can say in this issue.
> > > Cheers
> > > Tibor17
> > > 
> > > 
> > > 
> > > On Sat, Jul 20, 2019 at 12:25 PM Hervé BOUTEMY 
> > > 
> > > wrote:
> > > > little update on master branch runs in ASF Jenkins [1]:
> > > > build #240: stupidly failed on a Git issue on a node
> > > > 
> > > > then I launched build #241 *which worked ful

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-20 Thread Tibor Digana

Hello Herve,

no problem. I am investigating in
https://builds.apache.org/job/maven-box/job/maven/job/EOL/
the build is named after end-of-line because I have expected isues like
that. Only Jenkins Linux build fail, pls my last two commits in the branch
EOL.

My commits correspond to ".gitattributes" file and Jenkinsfile.
In ".gitattributes" I set eol=LF and I am archiving the folder
"core-it-suite" in Jenkinsfile.

It is strange that only Linux builds fail and thus there's is a kind of
platform dependency as it seems in this error.
I hope the archived ZIP file tells us more, e.g. logs at least.


Definitely something is wrong on Jenkins/Linux or in Jenkinsfile because
the build #22 was successful in
https://builds.apache.org/job/maven-box/job/maven/job/MNG-6294/ one month
ago, but I manually triggered #23 and I have got the same set of failed ITs
as we have in master today.

Cheers
Tibor


On Sat, Jul 20, 2019 at 7:25 PM Hervé BOUTEMY  wrote:

> I don't get where you want to go: there is no disagreement on reverting
> resolver from 1.4.0 to 1.3.3
>
> the question is, once this revert has been done, about the 4 ITs failing
> [2]:
> master #242 linux-jdk7 / Run ITs Linux Java 7
> 1. org.apache.maven.it
> .MavenITmng0553SettingsAuthzEncryptionTest.testitEncryption
> 2. org.apache.maven.it
> .MavenITmng0553SettingsAuthzEncryptionTest.testitBasic
> 3. org.apache.maven.it
> .MavenITmng4590ImportedPomUsesSystemPropertiesTest.testit
> 4. org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit
>
> I investigated these ITs and found one common fact: they try to define
> MAVEN_OPTS (and are the only ones during that)
> verifier.setEnvironmentVariable( "MAVEN_OPTS", "" );
> and this does not seem to be transferred to the verifier execution
>
> I don't know yet why this does not work on some Jenkins nodes, trying to
> add a few additional output to see if I can get some hints...
>
> any idea welcome.
>
> Regards,
>
> Hervé
>
> [2]
> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/master/242/testReport/
>
> Le samedi 20 juillet 2019, 12:48:27 CEST Tibor Digana a écrit :
> > I wrote in our chat that Maven Core fails randomly on random JDK versions
> > due to maven-resolver:1.4.0. It was proved by:
> >
> > + Jenkins on Maven Core
> > + Jira MNG-6714
> > + logical analysis of code in resolver.
> >
> > I wrote to Harald Wellmann that we should make code review together.
> > In my profession and company, the multithreading was my Java Advanced
> > expertize, so I am able to evaluate code which is thread safe and which
> is
> > not. Additionally, resolver:1.4.0 has logical problem even in single
> thread
> > which was proved by the branch where 11 unit tests failed on single
> thread
> > which of course must never happen.
> >
> > These things prove me to say that MRESOLVER-7 is not applicable in
> > production and we have to make code review together!
> >
> > Sorry to say, I can teach few devs with JSR-133 and thread safety
> regarding
> > Java Memory Model, and then the devs would probably understand why I
> found
> > issues in the code. Nevertheless still there are two problems with the
> > algorithm itself for whatever number of threads are executed:
> >
> > + one object is concurrently modified across multiple threads and the
> > object unexpectedly changes the state of the object
> > + there is one loop which resolves version but the last iteration wins,
> so
> > it loop looks to me ilogical and causes unnecessary object state
> > modifications
> >
> >
> > To be unbiased, all these things have to be clarified in PR on GitHub,
> > dicsussed, and changes have to be made in order to provide STABLE
> resolver.
> >
> > That's all I can say in this issue.
> > Cheers
> > Tibor17
> >
> >
> >
> > On Sat, Jul 20, 2019 at 12:25 PM Hervé BOUTEMY 
> >
> > wrote:
> > > little update on master branch runs in ASF Jenkins [1]:
> > > build #240: stupidly failed on a Git issue on a node
> > >
> > > then I launched build #241 *which worked fully*!!!
> > >
> > > to be sure of the stability, I relaunched the build, and build #242
> failed
> > > on
> > > the 4  ITs on Linux JDK 7: really strange. FYI, I'm personally on Linux
> > > with
> > > Java 7 and I can't reproduce the issue. And the fact that Maven 3.6.0
> > > branch
> > > fails on the same ITs, but sometimes on other configurations, makes me
> > > think
> > > there is an issue on some slave nodes on ASF Jenkins
> > >
> > > I integrated one little commit and build #243 still fails on the same 4
> > > ITs
> > >
> > >
> > > I suspect it's an issue on ASF Jenkins, but I can't prove anything: I
> > > can't
> > > even tell which slave node run successfully for build #241 but which
> ones
> > > failed on next builds...
> > >
> > > Any idea on how to investigate?
> > > Should this postpone the 3.6.2 release or not? To me, MNG-6712 fixed
> the
> > > real
> > > issue, that was causing infinite loops during artifacts resolution.
> > >
> > >
> > > A

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-20 Thread Hervé BOUTEMY

I don't get where you want to go: there is no disagreement on reverting 
resolver from 1.4.0 to 1.3.3

the question is, once this revert has been done, about the 4 ITs failing [2]:
master #242 linux-jdk7 / Run ITs Linux Java 7
1. 
org.apache.maven.it.MavenITmng0553SettingsAuthzEncryptionTest.testitEncryption
2. org.apache.maven.it.MavenITmng0553SettingsAuthzEncryptionTest.testitBasic
3. org.apache.maven.it.MavenITmng4590ImportedPomUsesSystemPropertiesTest.testit
4. org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit

I investigated these ITs and found one common fact: they try to define 
MAVEN_OPTS (and are the only ones during that)
verifier.setEnvironmentVariable( "MAVEN_OPTS", "" );
and this does not seem to be transferred to the verifier execution

I don't know yet why this does not work on some Jenkins nodes, trying to add a 
few additional output to see if I can get some hints...

any idea welcome.

Regards,

Hervé

[2] 
https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/master/242/testReport/

Le samedi 20 juillet 2019, 12:48:27 CEST Tibor Digana a écrit :
> I wrote in our chat that Maven Core fails randomly on random JDK versions
> due to maven-resolver:1.4.0. It was proved by:
> 
> + Jenkins on Maven Core
> + Jira MNG-6714
> + logical analysis of code in resolver.
> 
> I wrote to Harald Wellmann that we should make code review together.
> In my profession and company, the multithreading was my Java Advanced
> expertize, so I am able to evaluate code which is thread safe and which is
> not. Additionally, resolver:1.4.0 has logical problem even in single thread
> which was proved by the branch where 11 unit tests failed on single thread
> which of course must never happen.
> 
> These things prove me to say that MRESOLVER-7 is not applicable in
> production and we have to make code review together!
> 
> Sorry to say, I can teach few devs with JSR-133 and thread safety regarding
> Java Memory Model, and then the devs would probably understand why I found
> issues in the code. Nevertheless still there are two problems with the
> algorithm itself for whatever number of threads are executed:
> 
> + one object is concurrently modified across multiple threads and the
> object unexpectedly changes the state of the object
> + there is one loop which resolves version but the last iteration wins, so
> it loop looks to me ilogical and causes unnecessary object state
> modifications
> 
> 
> To be unbiased, all these things have to be clarified in PR on GitHub,
> dicsussed, and changes have to be made in order to provide STABLE resolver.
> 
> That's all I can say in this issue.
> Cheers
> Tibor17
> 
> 
> 
> On Sat, Jul 20, 2019 at 12:25 PM Hervé BOUTEMY 
> 
> wrote:
> > little update on master branch runs in ASF Jenkins [1]:
> > build #240: stupidly failed on a Git issue on a node
> > 
> > then I launched build #241 *which worked fully*!!!
> > 
> > to be sure of the stability, I relaunched the build, and build #242 failed
> > on
> > the 4  ITs on Linux JDK 7: really strange. FYI, I'm personally on Linux
> > with
> > Java 7 and I can't reproduce the issue. And the fact that Maven 3.6.0
> > branch
> > fails on the same ITs, but sometimes on other configurations, makes me
> > think
> > there is an issue on some slave nodes on ASF Jenkins
> > 
> > I integrated one little commit and build #243 still fails on the same 4
> > ITs
> > 
> > 
> > I suspect it's an issue on ASF Jenkins, but I can't prove anything: I
> > can't
> > even tell which slave node run successfully for build #241 but which ones
> > failed on next builds...
> > 
> > Any idea on how to investigate?
> > Should this postpone the 3.6.2 release or not? To me, MNG-6712 fixed the
> > real
> > issue, that was causing infinite loops during artifacts resolution.
> > 
> > 
> > As a side note, I started by doing a lot of cleanup on old merged
> > branches: it
> > would be nice if everybody did its own cleanup when merging
> > 
> > Regards,
> > 
> > Hervé
> > 
> > 
> > [1]
> > https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/
> > master/
> >  > /master/>> 
> > Le mardi 16 juillet 2019, 23:22:22 CEST Tibor Digana a écrit :
> > > Heads up!,
> > > 
> > > I am investigating the build errors on Maven Core.
> > > So I created three branches moved the HEAD backwards (i.e. git reset
> > 
> > --hard
> > 
> > > HEAD~12) and observed the outcome.
> > > 
> > > I have investigated 29 commits. Not sure how far to go...
> > > 
> > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11,
> > 
> > 12
> > 
> > > (16 ITs)
> > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> > > ITs)
> > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > > 
> > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > Always related to Linux.
> > 
> > > See the list of errors and bra

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-20 Thread Tibor Digana

I wrote in our chat that Maven Core fails randomly on random JDK versions
due to maven-resolver:1.4.0. It was proved by:

+ Jenkins on Maven Core
+ Jira MNG-6714
+ logical analysis of code in resolver.

I wrote to Harald Wellmann that we should make code review together.
In my profession and company, the multithreading was my Java Advanced
expertize, so I am able to evaluate code which is thread safe and which is
not. Additionally, resolver:1.4.0 has logical problem even in single thread
which was proved by the branch where 11 unit tests failed on single thread
which of course must never happen.

These things prove me to say that MRESOLVER-7 is not applicable in
production and we have to make code review together!

Sorry to say, I can teach few devs with JSR-133 and thread safety regarding
Java Memory Model, and then the devs would probably understand why I found
issues in the code. Nevertheless still there are two problems with the
algorithm itself for whatever number of threads are executed:

+ one object is concurrently modified across multiple threads and the
object unexpectedly changes the state of the object
+ there is one loop which resolves version but the last iteration wins, so
it loop looks to me ilogical and causes unnecessary object state
modifications


To be unbiased, all these things have to be clarified in PR on GitHub,
dicsussed, and changes have to be made in order to provide STABLE resolver.

That's all I can say in this issue.
Cheers
Tibor17



On Sat, Jul 20, 2019 at 12:25 PM Hervé BOUTEMY 
wrote:

> little update on master branch runs in ASF Jenkins [1]:
> build #240: stupidly failed on a Git issue on a node
>
> then I launched build #241 *which worked fully*!!!
>
> to be sure of the stability, I relaunched the build, and build #242 failed
> on
> the 4  ITs on Linux JDK 7: really strange. FYI, I'm personally on Linux
> with
> Java 7 and I can't reproduce the issue. And the fact that Maven 3.6.0
> branch
> fails on the same ITs, but sometimes on other configurations, makes me
> think
> there is an issue on some slave nodes on ASF Jenkins
>
> I integrated one little commit and build #243 still fails on the same 4 ITs
>
>
> I suspect it's an issue on ASF Jenkins, but I can't prove anything: I
> can't
> even tell which slave node run successfully for build #241 but which ones
> failed on next builds...
>
> Any idea on how to investigate?
> Should this postpone the 3.6.2 release or not? To me, MNG-6712 fixed the
> real
> issue, that was causing infinite loops during artifacts resolution.
>
>
> As a side note, I started by doing a lot of cleanup on old merged
> branches: it
> would be nice if everybody did its own cleanup when merging
>
> Regards,
>
> Hervé
>
>
> [1]
> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/
> master/
> 
>
> Le mardi 16 juillet 2019, 23:22:22 CEST Tibor Digana a écrit :
> > Heads up!,
> >
> > I am investigating the build errors on Maven Core.
> > So I created three branches moved the HEAD backwards (i.e. git reset
> --hard
> > HEAD~12) and observed the outcome.
> >
> > I have investigated 29 commits. Not sure how far to go...
> >
> > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11,
> 12
> > (16 ITs)
> > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> > ITs)
> > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> >
> > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > Always related to Linux.
> >
> > See the list of errors and branches:
> >
> >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-r
> > eset-head-12/1/#showFailuresLink
> >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3->
> reset-head-14/1/
> >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3->
> reset-head-29/1/
> >
> >
> >
> > From the MavenITmng0553SettingsAuthzEncryptionTest:
> >
> > [ERROR] Error executing Maven.
> > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> > java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml
> (No
> > such file or directory)
> > at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > (SecUtil.java:69)
> > at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> > at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> >
> > [ERROR] Failed to execute goal on project test: Could not resolve
> > dependencies for project
> > org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
> > dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT:
> > Failed to read artifact descriptor for
> > org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
> > artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to test
> > (http://localhost:32

Re: 3 ITs permanently fail on Linux / Maven Core

2019-07-20 Thread Hervé BOUTEMY

little update on master branch runs in ASF Jenkins [1]:
build #240: stupidly failed on a Git issue on a node

then I launched build #241 *which worked fully*!!!

to be sure of the stability, I relaunched the build, and build #242 failed on 
the 4  ITs on Linux JDK 7: really strange. FYI, I'm personally on Linux with 
Java 7 and I can't reproduce the issue. And the fact that Maven 3.6.0 branch 
fails on the same ITs, but sometimes on other configurations, makes me think 
there is an issue on some slave nodes on ASF Jenkins

I integrated one little commit and build #243 still fails on the same 4 ITs


I suspect it's an issue on ASF Jenkins, but I can't prove anything: I can't 
even tell which slave node run successfully for build #241 but which ones 
failed on next builds...

Any idea on how to investigate?
Should this postpone the 3.6.2 release or not? To me, MNG-6712 fixed the real 
issue, that was causing infinite loops during artifacts resolution.


As a side note, I started by doing a lot of cleanup on old merged branches: it 
would be nice if everybody did its own cleanup when merging

Regards,

Hervé


[1] https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/
master/

Le mardi 16 juillet 2019, 23:22:22 CEST Tibor Digana a écrit :
> Heads up!,
> 
> I am investigating the build errors on Maven Core.
> So I created three branches moved the HEAD backwards (i.e. git reset --hard
> HEAD~12) and observed the outcome.
> 
> I have investigated 29 commits. Not sure how far to go...
> 
> `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11, 12
> (16 ITs)
> `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> ITs)
> `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> 
> Always the ITs 0553, 4590, 4747 fail on several nodes.
> Always related to Linux.
> 
> See the list of errors and branches:
> 
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-r
> eset-head-12/1/#showFailuresLink
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-> 
> reset-head-14/1/
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-> 
> reset-head-29/1/
> 
> 
> 
> From the MavenITmng0553SettingsAuthzEncryptionTest:
> 
> [ERROR] Error executing Maven.
> org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml (No
> such file or directory)
> at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> (SecUtil.java:69)
> at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> 
> [ERROR] Failed to execute goal on project test: Could not resolve
> dependencies for project
> org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
> dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT:
> Failed to read artifact descriptor for
> org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
> artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to test
> (http://localhost:32917/): Not authorized
> 
> 
> 
> From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:
> 
> expected: d-29/test/core-it-suite/target/test-classes/mng-4590/pom.xml> but
> was: /test/core-it-suite/target/test-classes/mng-4590/${test.file}>
> 
> 
> 
> From the MavenITmng4747JavaAgentUsedByPluginTest:
> 
> junit.framework.AssertionFailedError
>   at
> org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit(MavenITm
> ng4747JavaAgentUsedByPluginTest.java:63)





-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

Re: 3 ITs permanently fail on Linux / Maven Core

14 matches

Site Navigation

Mail list logo

Footer information