It's not just merge jobs with random failures. Can anyone explain why this AR job failed: https://jenkins.opendaylight.org/releng/job/openflowplugin-validate-autorelease-oxygen/227/console ?
13:49:24 [INFO] ------------------------------------------------------------------------ 13:49:24 [INFO] Reactor Summary: 13:49:24 [INFO] 13:49:24 [INFO] mdsal-artifacts .................................... SUCCESS [ 35.062 s] 13:49:24 [INFO] mdsal-model-artifacts .............................. SUCCESS [ 34.991 s] 13:49:24 [INFO] config-artifacts ................................... SUCCESS [ 42.971 s] 13:49:24 [INFO] yang-test-plugin ................................... SUCCESS [01:13 min] 13:49:24 [INFO] mdsal-artifacts .................................... SUCCESS [ 43.020 s] 13:49:24 [INFO] autorelease-validate-projects ...................... SUCCESS [ 35.985 s] 13:49:24 [INFO] ------------------------------------------------------------------------ 13:49:24 [INFO] BUILD SUCCESS 13:49:24 [INFO] ------------------------------------------------------------------------ 13:49:24 [INFO] Total time: 01:41 min (Wall Clock) 13:49:24 [INFO] Finished at: 2018-02-06T13:49:24+00:00 13:49:24 [INFO] Final Memory: 51M/309M It all indicates success but still mysteriously fails. I don't see any reason why in the output. On Tue, Feb 6, 2018 at 8:45 AM, Thanh Ha <thanh...@linuxfoundation.org> wrote: > On Tue, Feb 6, 2018 at 5:20 AM, Robert Varga <n...@hq.sk> wrote: > >> On 06/02/18 06:53, Thanh Ha wrote: >> > The old-style mdsal-merge job deployed and I kicked off a build: >> > >> > https://jenkins.opendaylight.org/releng/view/Merge-Jobs/job/ >> mdsal-merge-oxygen/1/console >> > >> > It takes about 50 minutes for mdsal to build so we'll know in ~50 >> > minutes if that fixed the issue. >> >> Hello Thanh, >> >> the one thing I found is a difference in file upload strategy in the two >> jobs: >> >> old: >> >> > Deploying the main artifact iana-afn-safi-2013.07.04.12.0-SNAPSHOT.jar >> > Downloading: https://nexus.opendaylight.org >> /content/repositories/opendaylight.snapshot/org/opendaylight >> /mdsal/model/iana-afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml >> > Downloaded: https://nexus.opendaylight.org >> /content/repositories/opendaylight.snapshot/org/opendaylight >> /mdsal/model/iana-afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml >> (2 KB at 20.8 KB/sec) >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112.jar >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112.jar (23 KB at 2.4 KB/sec) >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112.pom >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112.pom (2 KB at 7.5 KB/sec) >> > Downloading: https://nexus.opendaylight.org >> /content/repositories/opendaylight.snapshot/org/opendaylight >> /mdsal/model/iana-afn-safi/maven-metadata.xml >> > Downloaded: https://nexus.opendaylight.org >> /content/repositories/opendaylight.snapshot/org/opendaylight >> /mdsal/model/iana-afn-safi/maven-metadata.xml (481 B at 31.3 KB/sec) >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/maven-metadata.xml >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 4.3 KB/sec) >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> maven-metadata.xml >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> maven-metadata.xml (481 B at 0.6 KB/sec) >> > Deploying the main artifact iana-afn-safi-2013.07.04.12.0- >> SNAPSHOT-javadoc.jar >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112-javadoc.jar >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112-javadoc.jar (50 KB at 58.1 KB/sec) >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/maven-metadata.xml >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 2.7 KB/sec) >> > Deploying the main artifact iana-afn-safi-2013.07.04.12.0- >> SNAPSHOT-sources.jar >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112-sources.jar >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013.07.04.12.0- >> 20180206.071107-112-sources.jar (13 KB at 13.0 KB/sec) >> > Uploading: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/maven-metadata.xml >> > Uploaded: https://nexus.opendaylight.org/content/repositories/opendayl >> ight.snapshot/org/opendaylight/mdsal/model/iana-afn-safi/ >> 2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 1.9 KB/sec) >> >> new: >> >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/maven-metadata.xml >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.jar.sha1 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.jar >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.jar.md5 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.pom >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.pom.sha1 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.pom.md5 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/maven-metadata.xml.sha1 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/maven-metadata.xml.md5 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-javadoc.jar >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111- >> javadoc.jar.sha1 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-javadoc.jar.md5 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-sources.jar >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111- >> sources.jar.sha1 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0-S >> NAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-sources.jar.md5 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/maven-metadata.xml >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/maven-metadata.xm >> l.sha1 >> > Uploading org/opendaylight/mdsal/model/iana-afn-safi/maven-metadata.xm >> l.md5 >> >> It seems we are doing things in different order, old: >> - download artifact-$VERSION metadata >> - upload artifact-$VERSION content (jar, pom) >> - download artifact metadata >> - upload artifact-$VERSION metadata >> - upload artifact metadata >> >> Whereas new does not seem to do any sort of ordering -- and uploads also >> checksums, etc. >> >> Now I don't know if nexus is doing anything the the metadata it has or >> how it reacts to parallel deployment of related artifacts. I would >> suggest creating a smarter strategy: >> >> - group files by their artifact name (i.e. directory) >> >> for each artifact file group: >> - upload content (jars, pom, javadoc, sources, etc. etc.) for an >> artifact in a serial manner first >> - upload versioned metadata >> - upload metadata >> >> Never use parallelism inside a group, use parallelism across groups -- >> i.e. deploy multiple artifacts concurrently while retaining file upload >> ordering matching how the old job does things within each artifact. >> >> Regards, >> Robert >> > > Hi Robert, > > So how the new maven merge job works is as follows: > > 1. Download maven-metadata.xml using wget recursive and copies it to > $WORKSPACE/m2repo and $WORKSPACE/m2repo-backup in the same structure as a > maven repo. > 2. Perform mvn clean deploy -DaltDeploymentRepository= > staging::default::file:$WORKSPACE/m2repo in this case Maven considers > this directory a m2repo and updates maven-metadata.xml as if the deployment > repo was on a webserver > 3. Diff the 2 repos m2repo and m2repo-backup and delete all files that > were not modified > 4. Use cURL + gnu-parallel to upload the artifacts to Nexus > > The issue is not with step 4 (Although uploading metadata last is a good > idea) but with step 2 (and I realize now why the validation tool in 3 > didn't catch some errors) when `mvn clean deploy` is running, for some mvn > occasionally produces artifacts and maven-metadata.xml that have the > different timestamps. This is is a known issue for sure with mvn 3.5.0 and > fixed in 3.5.2 I'm not sure if 3.3.9 had the issue or not. > > Last night Luis found a job that consistently always generated the wrong > metadata, when I switched that job to mvn35 it passed, would need more data > to confirm if 3.5.2 really solves the issue or we got lucky but it seems > like perhaps Maven 3.3.9 has issues with deploying to a local maven repo. > > As for why the validation tool failed to catch the issue was, we were only > checking for consistency inside of maven-metadata.xml so did not catch when > the maven-metadata.xml was consistent inside the file but still had > different filenames. > > This unfortunately puts us back on the old merge job which causes Jenkins > instability but considering our crunch time now, a little instability with > Jenkins is better than having bad metadata. > > Thanh > > _______________________________________________ > controller-dev mailing list > controller-dev@lists.opendaylight.org > https://lists.opendaylight.org/mailman/listinfo/controller-dev > >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
