Hi,
David, can you have a look at this: https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ <https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/> This job builds a local Maven repo (unsigned) on request (somebody with Jenkins access have to trigger it): https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/ I will now change this that the above folder is publsihed as a separate folder (also including build number) to ASF’s snapshot website. Does this directory structure (below maven-local) looks fine? ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de From: Uwe Schindler <u...@thetaphi.de> Sent: Tuesday, May 25, 2021 3:06 PM To: dev@solr.apache.org Subject: RE: Proposal to pin the Lucene snapshot version on main Hi, sorry for not acting faster. I was a bit busy with new kitchen, my main work, vaccination,… The main problem was that I was not able to sign artifacts from Jenkins, but with Alan Woodward’s tip on lucene’s dev list regarding the disabled artifact signing (I would not think that we really need this here…) I am confident. Problem is now how to synchronize the whole repository (so keep older artifacts, the current Lucene jobs creates a completely new local Maven artifact repository).When you sync this to the snapshot website, older Lucene preview releases would get lost. Maybe it’s better to store it somewhere inside jenkins. An alternative is to create a separate repository and then update base URL and artifact version inside the gradle scripts. So it could be like a repository with the jenkins build number in the repo’s base URL (needs to be cahnged in the remotes of Gradle) and the version number. IMHO, this would be a 2 line change instead of a one-line change inside solr’s build.gradle, so it’s not problematic. Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de <mailto:u...@thetaphi.de> From: David Smiley <dsmi...@apache.org <mailto:dsmi...@apache.org> > Sent: Tuesday, May 25, 2021 2:01 PM To: Uwe Schindler <u...@thetaphi.de <mailto:u...@thetaphi.de> > Cc: dev@solr.apache.org <mailto:dev@solr.apache.org> ; Cassandra Targett <casstarg...@gmail.com <mailto:casstarg...@gmail.com> > Subject: Re: Proposal to pin the Lucene snapshot version on main Thanks again for volunteering Uwe. I'm wondering what the status of this is? There was recently a major change in Lucene to move SpanQueries, and I think we're going to feel it in Solr. If not, it's just a matter of time before there is some other impactful change. ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Fri, May 7, 2021 at 10:22 AM Uwe Schindler < <mailto:u...@thetaphi.de> u...@thetaphi.de> wrote: Hi, Sure! I will play a little bit around. If needed, I can add some property to the lucene build to locate a "local repo" (I think we have that already). I just have to figure out, if some local repo, copied to some Webserver works well as a full M2 repository, especially how to handle multiple intermediary releases in it. Uwe Am May 7, 2021 1:47:45 PM UTC schrieb David Smiley < <mailto:dsmi...@apache.org> dsmi...@apache.org>: Sounds great Uwe! Can you do it please? ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Fri, May 7, 2021 at 1:55 AM Uwe Schindler < <mailto:u...@thetaphi.de> u...@thetaphi.de> wrote: Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only manually triggered. It builds the maven artifacts into a local file system repo. As post build job we publish the output folder on nighties repo like the ref guide. The version number is passed as -DreleaseVersion system property (e.g. populated by Jenkins from the build number), like we do on artifacts builds on jenkins anyways. It's no snapshot anymore, but like an half official build with some custom version internal to our process. If you want to upgrade Solr's Lucene, just press "build" on jenkins, wait for it to finish, and finally copy-paste the version number from Jenkins build description into gradle dependency version file. Uwe Uwe Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett < <mailto:casstarg...@gmail.com> casstarg...@gmail.com>: It could be a manual process if that’s what you think is easiest and it could still be on nightlies. It’s just a curl command to push something up: <https://nightlies.apache.org/authoring.html> https://nightlies.apache.org/authoring.html. If it is a manual process, I think it would be better for it to be hosted in a place any of us could access, in case you’re busy/something happens to you and we need to pin a different snapshot. I’m not sure I see a reason why Solr couldn’t have our own Jenkins job that built Lucene once a week (or on demand, or whatever cadence that works) and pushes the snapshot to nightlies - we don’t have to use one of Lucene’s jobs, do we? I feel like we could also cascade a new Lucene snapshot build into a “Lucene validation” Solr build that verifies everything is OK with a new snapshot before updating the hosted snapshot on nightlies (although, I don’t actually know how to do that, it just seems like something that could be done). Cassandra On May 6, 2021, 10:13 AM -0500, David Smiley < <mailto:dsmi...@apache.org> dsmi...@apache.org>, wrote: I asked on the Lucene dev list about possible use of the Nightlies server. We don't want to pollute nightlies on a regular basis (I think); this would be an on-demand thing. As such, I'm not sure a CI server is the best way to approach this vs a manual script publishing to an ASF personal Home space. ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Mon, May 3, 2021 at 10:55 AM Cassandra Targett < <mailto:casstarg...@gmail.com> casstarg...@gmail.com> wrote: I’m sort of surprised no one has mentioned the <https://nightlies.apache.org/> https://nightlies.apache.org/ server, which could be used for this purpose. Jenkins can push to it, and nothing gets deleted until we decide to delete it (or overwrite it). Solr Operator builds go there as do drafts of the Ref Guide pre-publication (in different directories now, but we’ll fix that eventually). Cassandra On May 3, 2021, 9:15 AM -0500, David Smiley < <mailto:dsmi...@apache.org> dsmi...@apache.org>, wrote: RE "an external system like Maven" -- we're merely talking about adding another JAR repo to the list of repos we already have. Heck, for this limited purpose, we could even use <http://home.apache.org/~dsmiley/> http://home.apache.org/~dsmiley/ Note that the Lucene project already uses home dirs of some users for benchmark data. If we were talking about adding a Maven *build* then I would totally appreciate your concern. I volunteer to use my space for this purpose. ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya < <mailto:ichattopadhy...@gmail.com> ichattopadhy...@gmail.com> wrote: > Let’s not complicate things. By bringing in external systems like Maven, I think we're complicating things even though a straight forward way (git submodules) exists. On Sat, May 1, 2021 at 5:00 PM Jan Høydahl < <mailto:janhoy-apa...@cominvent.com> janhoy-apa...@cominvent.com> wrote: Sub modules is for organizing internal repos, not for pulling in external deps. Let’s not complicate things. And once we switch main to 10.x we’d need to use pure jar dependencies in branch_9x to depend upon actually released and voted Lucene binaries. We need some tooling to smoothly work with bleeding edge Lucene including local snapshot builds with not yet pushed changes to Lucene. I’d rather put in some work to establish such tooling or procedures. Jan Høydahl 1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya < <mailto:ichattopadhy...@gmail.com> ichattopadhy...@gmail.com>: > What submodules don't solve is releases - if you're > on a particular unreleased Lucene version then releasing > Solr would still mean you need some kind of > "public" pinned Lucene release for the > Mavenworld. Can we then update the submodule to point to the release tag or sha that Lucene got released from? On Sat, 1 May, 2021, 11:45 am Dawid Weiss, < <mailto:dawid.we...@gmail.com> dawid.we...@gmail.com> wrote: > Other than literally adding the git submodule, would we do anything else to > modify the gradle build so that or do we (and Jenkinsfile) have to manually > do a step to install Lucene before proceeding? Technically you add a submodule and then make a composite build from Solr side. I can provide a PR that does it if you wish... This is simple. What submodules don't solve is releases - if you're on a particular unreleased Lucene version then releasing Solr would still mean you need some kind of "public" pinned Lucene release for the Maven world. If you distribute the entire thing as binaries you can just publish a snapshot of Lucene code, of course. > I think adding git submodule means that we have to add back in all the build > code for it. No. The submodule is just a reference to a particular repository (and commit) - the build code and remains nearly identical as it was with Lucene. Composite builds in gradle are quite nice in that they're (almost) transparent compared to dependency references. > At which point, I'd rather just copy and commit the > code they have so we don't have to learn another git system. It's not really a different git system - it's a mechanism of interacting with multiple repositories at once. Yes, it does add some additional complexity but I think sooner or later you'll have to interact with it anyway - many people favor monorepos these days and this is a common way to assemble them from fragmented sub-repositories. > I've > heard submodules don't play nice with Jenkins, but don't have any > direct experience with them. Some tools may require an additional flag to initialize and clone sub-modules on the initial pull. Or manually. git submodule init git submodule update --recursive. To summarize - I am not pushing for submodules, I do think they're a viable alternative but pinned Maven references are simpler. The problem with maven references is that we'd need a long(er) term maven repository where such pinned references would live (without being wiped out). Perhaps infra can help here and it'd solve the problem. Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: <mailto:dev-unsubscr...@solr.apache.org> dev-unsubscr...@solr.apache.org For additional commands, e-mail: <mailto:dev-h...@solr.apache.org> dev-h...@solr.apache.org -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
