Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only manually triggered. It builds the maven artifacts into a local file system repo. As post build job we publish the output folder on nighties repo like the ref guide.
The version number is passed as -DreleaseVersion system property (e.g. populated by Jenkins from the build number), like we do on artifacts builds on jenkins anyways. It's no snapshot anymore, but like an half official build with some custom version internal to our process. If you want to upgrade Solr's Lucene, just press "build" on jenkins, wait for it to finish, and finally copy-paste the version number from Jenkins build description into gradle dependency version file. Uwe Uwe Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett <[email protected]>: >It could be a manual process if that’s what you think is easiest and it >could still be on nightlies. It’s just a curl command to push something >up: https://nightlies.apache.org/authoring.html. If it is a manual >process, I think it would be better for it to be hosted in a place any >of us could access, in case you’re busy/something happens to you and we >need to pin a different snapshot. > >I’m not sure I see a reason why Solr couldn’t have our own Jenkins job >that built Lucene once a week (or on demand, or whatever cadence that >works) and pushes the snapshot to nightlies - we don’t have to use one >of Lucene’s jobs, do we? > >I feel like we could also cascade a new Lucene snapshot build into a >“Lucene validation” Solr build that verifies everything is OK with a >new snapshot before updating the hosted snapshot on nightlies >(although, I don’t actually know how to do that, it just seems like >something that could be done). > >Cassandra >On May 6, 2021, 10:13 AM -0500, David Smiley <[email protected]>, >wrote: >> I asked on the Lucene dev list about possible use of the Nightlies >server. We don't want to pollute nightlies on a regular basis (I >think); this would be an on-demand thing. As such, I'm not sure a CI >server is the best way to approach this vs a manual script publishing >to an ASF personal Home space. >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> >> >> > On Mon, May 3, 2021 at 10:55 AM Cassandra Targett ><[email protected]> wrote: >> > > I’m sort of surprised no one has mentioned >the https://nightlies.apache.org/ server, which could be used for this >purpose. Jenkins can push to it, and nothing gets deleted until we >decide to delete it (or overwrite it). Solr Operator builds go there as >do drafts of the Ref Guide pre-publication (in different directories >now, but we’ll fix that eventually). >> > > >> > > Cassandra >> > > On May 3, 2021, 9:15 AM -0500, David Smiley <[email protected]>, >wrote: >> > > > RE "an external system like Maven" -- we're merely talking >about adding another JAR repo to the list of repos we already have. >Heck, for this limited purpose, we could even >use http://home.apache.org/~dsmiley/ Note that the Lucene project >already uses home dirs of some users for benchmark data. If we were >talking about adding a Maven *build* then I would totally appreciate >your concern. >> > > > >> > > > I volunteer to use my space for this purpose. >> > > > >> > > > ~ David Smiley >> > > > Apache Lucene/Solr Search Developer >> > > > http://www.linkedin.com/in/davidwsmiley >> > > > >> > > > >> > > > > On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya ><[email protected]> wrote: >> > > > > > > Let’s not complicate things. >> > > > > > By bringing in external systems like Maven, I think we're >complicating things even though a straight forward way (git submodules) >exists. >> > > > > > >> > > > > > > On Sat, May 1, 2021 at 5:00 PM Jan Høydahl ><[email protected]> wrote: >> > > > > > > > Sub modules is for organizing internal repos, not for >pulling in external deps. Let’s not complicate things. And once we >switch main to 10.x we’d need to use pure jar dependencies in branch_9x >to depend upon actually released and voted Lucene binaries. >> > > > > > > > >> > > > > > > > We need some tooling to smoothly work with bleeding >edge Lucene including local snapshot builds with not yet pushed changes >to Lucene. I’d rather put in some work to establish such tooling or >procedures. >> > > > > > > > >> > > > > > > > Jan Høydahl >> > > > > > > > >> > > > > > > > > 1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya ><[email protected]>: >> > > > > > > > > >> > > > > > > > > > What submodules don't solve is releases - if you're >> > > > > > > > > > on a >> > > > > > > > > particular unreleased Lucene version then releasing >> > > > > > > > > > Solr would still mean you need some kind of >> > > > > > > > > > "public" pinned Lucene release for the >> > > > > > > > > > Mavenworld. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > Can we then update the submodule to point to the >release tag or sha that Lucene got released from? >> > > > > > > > > > >> > > > > > > > > > On Sat, 1 May, 2021, 11:45 am Dawid Weiss, ><[email protected]> wrote: >> > > > > > > > > > > > Other than literally adding the git submodule, >would we do anything else to modify the gradle build so that or do we >(and Jenkinsfile) have to manually do a step to install Lucene before >proceeding? >> > > > > > > > > > > >> > > > > > > > > > > Technically you add a submodule and then make a >composite build from >> > > > > > > > > > > Solr side. I can provide a PR that does it if you >wish... This is >> > > > > > > > > > > simple. What submodules don't solve is releases - >if you're on a >> > > > > > > > > > > particular unreleased Lucene version then >releasing Solr would still >> > > > > > > > > > > mean you need some kind of "public" pinned Lucene >release for the >> > > > > > > > > > > Maven world. If you distribute the entire thing >as binaries you can >> > > > > > > > > > > just publish a snapshot of Lucene code, of >course. >> > > > > > > > > > > >> > > > > > > > > > > > I think adding git submodule means that we have >to add back in all the build code for it. >> > > > > > > > > > > >> > > > > > > > > > > No. The submodule is just a reference to a >particular repository (and >> > > > > > > > > > > commit) - the build code and remains nearly >identical as it was with >> > > > > > > > > > > Lucene. Composite builds in gradle are quite nice >in that they're >> > > > > > > > > > > (almost) transparent compared to dependency >references. >> > > > > > > > > > > >> > > > > > > > > > > > At which point, I'd rather just copy and commit >the >> > > > > > > > > > > > code they have so we don't have to learn >another git system. >> > > > > > > > > > > >> > > > > > > > > > > It's not really a different git system - it's a >mechanism of >> > > > > > > > > > > interacting with multiple repositories at once. >Yes, it does add some >> > > > > > > > > > > additional complexity but I think sooner or later >you'll have to >> > > > > > > > > > > interact with it anyway - many people favor >monorepos these days and >> > > > > > > > > > > this is a common way to assemble them from >fragmented >> > > > > > > > > > > sub-repositories. >> > > > > > > > > > > >> > > > > > > > > > > > I've >> > > > > > > > > > > > heard submodules don't play nice with Jenkins, >but don't have any >> > > > > > > > > > > > direct experience with them. >> > > > > > > > > > > >> > > > > > > > > > > Some tools may require an additional flag to >initialize and clone >> > > > > > > > > > > sub-modules on the initial pull. Or manually. >> > > > > > > > > > > >> > > > > > > > > > > git submodule init >> > > > > > > > > > > git submodule update --recursive. >> > > > > > > > > > > >> > > > > > > > > > > To summarize - I am not pushing for submodules, I >do think they're a >> > > > > > > > > > > viable alternative but pinned Maven references >are simpler. The >> > > > > > > > > > > problem with maven references is that we'd need a >long(er) term maven >> > > > > > > > > > > repository where such pinned references would >live (without being >> > > > > > > > > > > wiped out). Perhaps infra can help here and it'd >solve the problem. >> > > > > > > > > > > >> > > > > > > > > > > Dawid >> > > > > > > > > > > >> > > > > > > > > > > >--------------------------------------------------------------------- >> > > > > > > > > > > To unsubscribe, e-mail: >[email protected] >> > > > > > > > > > > For additional commands, e-mail: >[email protected] >> > > > > > > > > > > -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
