Re: Weirdness in jenkins build (tests not isolated as expected)
Gotcha! - Sijie On Mon, Jun 4, 2018 at 11:50 AM, Ivan Kelly wrote: > >> Two concurrent builds on the same machine will have different > >> workspaces (@2 or @3 will be appended to $WORKSPACE). > > > > I am not sure about that. If taking a look at the jenkins job output, the > > workspace is " > > /home/jenkins/jenkins-slave/workspace/pulsar_precommit_integrationtests" > > It only adds the @ for the second concurrent build onwards. > > https://github.com/jenkinsci/jenkins/blob/master/core/src/ > main/java/hudson/slaves/WorkspaceList.java#L181 > > -Ivan >
Re: Weirdness in jenkins build (tests not isolated as expected)
>> Two concurrent builds on the same machine will have different >> workspaces (@2 or @3 will be appended to $WORKSPACE). > > I am not sure about that. If taking a look at the jenkins job output, the > workspace is " > /home/jenkins/jenkins-slave/workspace/pulsar_precommit_integrationtests" It only adds the @ for the second concurrent build onwards. https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/slaves/WorkspaceList.java#L181 -Ivan
Re: Weirdness in jenkins build (tests not isolated as expected)
On Mon, Jun 4, 2018 at 4:42 AM, Ivan Kelly wrote: > > 1) the job is using `$WORKSPACE/.repository` as the maven local repo. > > 2) concurrent build is enabled > > 3) it is a freestyle build, including two maven runs. > > Two concurrent builds on the same machine will have different > workspaces (@2 or @3 will be appended to $WORKSPACE). I am not sure about that. If taking a look at the jenkins job output, the workspace is " /home/jenkins/jenkins-slave/workspace/pulsar_precommit_integrationtests" > We should > disable concurrent build on the same machine, as they'll interfere > with each other's docker images, so we wouldn't be testing what we > think we're testing. > > -Ivan >
Re: Weirdness in jenkins build (tests not isolated as expected)
> 1) the job is using `$WORKSPACE/.repository` as the maven local repo. > 2) concurrent build is enabled > 3) it is a freestyle build, including two maven runs. Two concurrent builds on the same machine will have different workspaces (@2 or @3 will be appended to $WORKSPACE). We should disable concurrent build on the same machine, as they'll interfere with each other's docker images, so we wouldn't be testing what we think we're testing. -Ivan
Re: Weirdness in jenkins build (tests not isolated as expected)
Not fully sure, but I guess the problem comes from concurrent build and two maven runs in integration tests. https://github.com/apache/incubator-pulsar/blob/master/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy#L29 https://github.com/apache/incubator-pulsar/blob/master/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy#L38 1) the job is using `$WORKSPACE/.repository` as the maven local repo. 2) concurrent build is enabled 3) it is a freestyle build, including two maven runs. if two *concurrent* jobs are interleaving, the second build (mvn install) might overwrite the first build's built artifacts before first build runs its tests. On Mon, Jun 4, 2018 at 3:02 AM, Ivan Kelly wrote: > Hi folks, > > I've just come across something very strange in a jenkins build that > others should be aware of in case they hit across the same thing. > > https://builds.apache.org/job/pulsar_precommit_integrationtests/1782/ > > This build is a bulld of the branch > 69b5976d8b40789f7f139185ffc1e5cc0bbe4f4b > https://github.com/ivankelly/incubator-pulsar/tree/ > 69b5976d8b40789f7f139185ffc1e5cc0bbe4f4b > > It failed on the command > > timeout 60 bash -c until test -S /var/run/supervisor/supervisor.sock; > do sleep 0.1; done > > https://builds.apache.org/job/pulsar_precommit_integrationtests/1782/ > artifact/tests/integration/smoke/target/surefire-reports/ > org.apache.pulsar.tests.integration.TestSmoke-output.txt > > Now, the command it failed on isn't what is interesting (I'm still > working on this stuff), but what is interesting it that the timeout is > not part of 69b5976. It was a patch I submitted in parallel. > > The place it should be picking it up from the classpath is > /home/jenkins/jenkins-slave/workspace/pulsar_precommit_ > integrationtests/tests/integration-tests-utils/target/classes, > and this cleaned and installed into earlier in the test. > > So, somehow maven is picking up artifacts from somewhere else, and I'm > stumped to know where. > > -ivan >
