Is the release of the docker image going to be part of the standard Lucene/Solr release process? I ask because I’m planning on starting an 8.5 release next week, and I know nothing about Docker images...
> On 25 Feb 2020, at 16:04, David Smiley <david.w.smi...@gmail.com> wrote: > > I wonder what official information ASF provides on this matter. I did some > searching and found this page: > https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines > <https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines>. > (see "Docker" heading) which was nicely short and to the point but doesn't > seem to answer. "DRAFT" 4 times is at the top of this page. And it doesn't > address all questions. I wonder about Houston's point as well; I'm not sure > we can simply update an image just because the JAR files didn't change. > Maybe; maybe not. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > <http://www.linkedin.com/in/davidwsmiley> > > On Tue, Feb 25, 2020 at 10:35 AM Jan Høydahl <jan....@cominvent.com > <mailto:jan....@cominvent.com>> wrote: > I think se are not technically re-releasing Solr 8.4 even if that official > docker image gets re-built with latest versions of Ubuntu and JRE11 when re > release e.g. 8.5. > The Apache Solr/Lucene binaries are still the exact same bits, we just change > the base image — equivalent to upgrading Linux and Java on physical servers. > Of course there could be bugs manifested with a certain combination of Linux > + JRE + Solr that potentially would cause solr:x.y to break further down the > road, that the simple shell tests run during release might not catch. > > Jan > >> 25. feb. 2020 kl. 16:13 skrev Houston Putman <houstonput...@gmail.com >> <mailto:houstonput...@gmail.com>>: >> >> I have a separate question about the release process. As I currently >> understand it, whenever docker-solr is released, every version in its >> configs >> <https://github.com/docker-solr/docker-solr/blob/40857941a5850b17906a61cad0ff58278f47e589/.travis.yml#L22> >> is rebuilt and re-released. This means that versions of the docker-solr >> images are not necessarily concrete, whereas the versions of solr are very >> concrete. >> >> I imagine that by taking over the docker image as an official Apache image, >> this re-releasing of versions will no longer be allowed. That makes me think >> that adding a docker publishing step in the release process is necessary. >> There will also need to be extensive testing of that docker image in that >> process because we won't be able to retroactively fix issues anymore. >> >> If Apache is more relaxed about re-releasing the same version, then this is >> less of an issue. >> >> - Houston >> >> On Thu, Feb 13, 2020 at 9:42 AM Jan Høydahl <jan....@cominvent.com >> <mailto:jan....@cominvent.com>> wrote: >> I propose we continue to work with the existing docker-solr repo for some >> time still, until we fully understand how we want to proceed with moving to >> ASF owned git infra and hub accounts. >> >> I feel that some work should have higher priority for now: >> - Document running Solr on Docker in Ref Guide >> - Start thinking about how to include Docker image publishing in the release >> process >> - Adding a simplistic Dockerfile to our main git repo and a gradle task for >> building >> - Update the README in docker-solr repo to reflect the new ownership >> >> Some of these could be sub tasks of SOLR-14168. >> >> Other thoughts? >> >> Jan >> >>> 12. jan. 2020 kl. 04:46 skrev David Smiley <david.w.smi...@gmail.com >>> <mailto:david.w.smi...@gmail.com>>: >>> >>> > Yes, it should be easy to build a docker image «from source», or at least >>> > as a gradle build task. That could piggy-back on the distro tgz file >>> > which should make it not too different - we just pull the release from >>> > local disk instead of from the mirrors. >>> >>> We do this at Salesforce in our local Lucene_Solr fork to also produce a >>> docker image. It's not a big deal but I could share it if we want to >>> consider going this direction. It's kinda necessary if we want to release >>> this all at once instead of requiring a 'tgz' be released first, which in >>> turn somewhat requires some signatures of that binary that then become >>> irrelevant to check when producing the Docker image. It's also super nice >>> for those who fork Solr to also produce a Docker image easily (like us). >>> >>> ~ David Smiley >>> Apache Lucene/Solr Search Developer >>> http://www.linkedin.com/in/davidwsmiley >>> <http://www.linkedin.com/in/davidwsmiley> >>> >>> On Sat, Jan 11, 2020 at 5:45 PM Jan Høydahl <jan....@cominvent.com >>> <mailto:jan....@cominvent.com>> wrote: >>>>> 1. Are we allowed to maintain ASF code in a non-ASF repo? If not, how do >>>>> we transition to >>>>> an ASF git repo? >>>>> * Can it be a sub folder in our main repo or does it need to be a >>>>> separate repo? >>>> >>>> The way it works (from the official library’s point of view), is that we >>>> maintain >>>> https://github.com/docker-library/official-images/blob/master/library/solr >>>> <https://github.com/docker-library/official-images/blob/master/library/solr> >>>> which contains a link to a repo (in our case >>>> https://github.com/docker-solr/docker-solr.git >>>> <https://github.com/docker-solr/docker-solr.git>) and particular git >>>> commit, and a particular directory for different versions. That is >>>> consumed by their build infrastructure. The library team reviews changes >>>> we make to that file, and the corresponding changes we made to the >>>> Dockerfiles and bash scripts in the docker-solr repo, so it needs to be >>>> readily available and it needs to be easy to see what has changed. >>>> >>>> I think one could theoretically move this into the main Solr repo and >>>> point to its GitHub address, but that would make things slower and much >>>> harder to review. So I think it’s much better to keep the separate repo. I >>>> briefly looked for some official guidance on this, but couldn’t find it >>>> spelled out explicitly. I did see >>>> https://github.com/docker-library/official-images#maintainership >>>> <https://github.com/docker-library/official-images#maintainership> which >>>> talks about maintaining git history. >>>> Note also that I already use a “docker-solr” GitHub org for the repo, >>>> rather than my own account, to make it easier to vary ownership. >>>> >>>> If you are dead-set to put it into the main repo, I’d run that discussion >>>> past the library team first before sinking engineering time. >>> >>> I just discovered https://hub.docker.com/u/apache >>> <https://hub.docker.com/u/apache> - which is Apache’s own docker org. I see >>> some images there are hosted in separate apache git repos, example CouchDB: >>> https://github.com/apache/couchdb-docker >>> <https://github.com/apache/couchdb-docker> pushed to >>> https://hub.docker.com/r/apache/couchdb >>> <https://hub.docker.com/r/apache/couchdb> - and >>> https://hub.docker.com/_/couchdb <https://hub.docker.com/_/couchdb> >>> (official). The source of both hub locations seems to be the same >>> apache/couchdb-docker git repo. I see that the person who files PRs aginst >>> the official image repo is Joan Touzet (http://people.apache.org/~wohali/ >>> <http://people.apache.org/~wohali/>) who is a CouchDB committer. Perhpas >>> this is a model for us to follow. >>> >>> We may also want to consult LEGAL-503 >>> <https://issues.apache.org/jira/browse/LEGAL-503?focusedCommentId=17003438&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17003438> >>> where the Beam project asked a similar question a few weeks ago, and the >>> reply is: >>> >>> if you would like to continue linking to the Docker release artifact from >>> the https://beam.apache.org <https://beam.apache.org/> you will have: >>> 1. Transition to the official ASF dockerhub org: >>> https://hub.docker.com/u/apache <https://hub.docker.com/u/apache> >>> 2. Start including that binary convenience artifact into your VOTE threads >>> on Beam releases >>> 3. Make sure that all Cat-X licenses are ONLY brought into your container >>> via FROM statements >>> >>> So bullet point #1 there answers this question. Regarding point #2 and #3 >>> see below. >>> >>>>> 2. How will the current build/test/publish process need to change? >>>>> * Can we continue using travis for CI? >>>> >>>> In the short term, sure. >>>> >>>> Travis has been great for us — it is free, it builds fast enough, the UI >>>> is nice, the config is simple, the integration is good, and support was >>>> helpful. >>>> Last year Travis CI got acquired, followed by layoffs of senior >>>> engineering staff, so there are concerns about its future, but nothing has >>>> really changed to affect us. >>>> >>>> I imagine it would be nicer to have it in the normal Apache Jenkins world, >>>> but I’m not volunteering for that migration. :-) >>>> >>>> If we want to stay on Travis, there may be some configuration changes >>>> required (roles/permissions/credentials and such that are tied to my >>>> account). >>>> >>>> Oh and just to make it clear: the CI does 2 things: >>>> - it sets build status on GitHub commits (although there is currently no >>>> enforcement to allow only passing PRs to be merged or things like that, or >>>> have review/automerge workflows which would be nice to have) >>>> - and it pushes builds to the >>>> https://hub.docker.com/repository/docker/dockersolr/docker-solr >>>> <https://hub.docker.com/repository/docker/dockersolr/docker-solr> repo — >>>> but those are only used for testing, they are not the docker images that >>>> provide the official images. I've found that occasionally useful, but we >>>> could decide to not do that, or do it differently within the Apache >>>> infrastructure. >>> >>> So I see other ASF projects using travis as well, perhaps ASF has an >>> account/license? If we continue to use it or if we migrate to Jenkins, we >>> either way need to run the build and test and then push builds to the >>> Apache Docker Hub repository space (making the image pull’able with docker >>> pull apache/solr:tag >>> The actual producing of official image will be yet another PR to the docker >>> owned official-images repo. >>> >>>>> * Should publishing of new Docker be a RM responsibility, or >>>>> something that happens right >>>>> after each release like the ref-guide? >>>> >>>> I don’t have a strong opinion. I typically tried to do it as soon as I >>>> became aware of a new version via the solr-user mailing list or twitter. >>>> Sometimes same day, sometimes it would take a week because of changes I >>>> need to make or extra things I wanted to do. >>>> But if I’m more than a few days late someone would be asking about it :-) >>>> The official library team review is usually very fast, same day or 24h. >>> >>> See point #2 from LEGAL-503 above. If we want to officially document / >>> endorse / link to the image on hub we may want to include the docker image >>> in the VOTE. I see that the Beam project includes this in their >>> release-guide (publishing SDK images): >>> https://beam.apache.org/contribute/release-guide/ >>> <https://beam.apache.org/contribute/release-guide/>. What they do is that >>> push a RC tagged version to their docker-hub as part of the release and >>> include it in the VOTE. >>> >>>>> 3. Legal stuff - when we as a project file a PR to update the official >>>>> solr docker images, >>>>> are we then legally releasing a binary version of Solr? >>>>> Technically it is Docker CI that build and publish the images, we >>>>> just initiate it… >>>> >>>> I don’t know about that (or how that matters?) >>> >>> Oh, legal stuff matters a lot for Apache :) Again, I think LEGAL-503 >>> answers this. Bullet #3 there requres the project to make sure that our >>> Dockerfile does not bring in Cat-X licensed software into the Docker layers >>> built by us. Since we base our image on the ‘openjdk’ base image, which >>> contains GNU/Linux binaries and the JDK, the only things we'd need to >>> verify is what we bring into our Docker layers through apt-get, wget etc. >>> Below is a list of what I found: >>> >>> acl - GPL - provides tool setfacl, used only in tests, can be removed? >>> dirmngr, gpg - GPL - used only during docker build phase, may be apt >>> install and uninstalled in the same RUN command >>> lsof - BSD license >>> procps - GPL - provides the ‘ps’ command needed by bin/solr. This is part >>> of openjdk:11 but not openjdk:11-slim... >>> wget - GPL - used during build only, can be uninstalled after use >>> netcat - PublicDomain >>> gosu - GPL - can be removed or replaced with su-exec (MIT) >>> tini - MIT >>> >>>>> Do we know any other ASF project that maintain their own official >>>>> docker image? >>>> >>>> I've looked at >>>> https://github.com/docker-library/official-images/tree/master/library >>>> <https://github.com/docker-library/official-images/tree/master/library> >>>> and spotted https://github.com/carlossg/docker-maven >>>> <https://github.com/carlossg/docker-maven> which is maintained by an >>>> Apache committer. >>> >>> So couchDB is another example. And there are so many other projects in >>> Apache’s docker-hub org that I suppose there may be others. >>> >>>> Marcus wrote: >>>> >>>>> I think that regardless of what the community decides to do with the >>>>> docker-solr repo, a good first step would be to add a Docker folder to the >>>>> Apache repository that contains a base Dockerfile and a README. In that >>>>> README, users can be directed to the location of the docker-solr repo, >>>>> wherever that may be, or leverage the Dockerfile in the Apache repo as a >>>>> starting point for building their own image. >>>> >>>> >>>> I think that could be useful; but it then does start to become messy >>>> almost immediately: Users will expect these self-built images and the >>>> official images to work the same, and given that docker-solr has various >>>> extra scripts (eg to create collections at startup), you’d then have to >>>> copy them into the repo (and now have duplicate maintenance, need to test >>>> them). Or you could explicitly decide not to do that, but then your users >>>> will be asking how to achieve the same functionality with their images. >>>> >>>> I would address this as a separate issue. Let’s get the existing image >>>> flow taken care of first. >>> >>> Yes, it should be easy to build a docker image «from source», or at least >>> as a gradle build task. That could piggy-back on the distro tgz file which >>> should make it not too different - we just pull the release from local disk >>> instead of from the mirrors. >>> >>> I also saw some projects that have Jenkins routinely publish SNAPSHOT >>> releases to docker-hub, see e.g. >>> https://hub.docker.com/r/apache/syncope/tags >>> <https://hub.docker.com/r/apache/syncope/tags> which is also nice if we >>> want to have people test out things with unreleased versions or master >>> branch, then it is always only a docker run command away :) >>> >>> Well, I hope other committers also join this discussion and bring perhaps >>> other points of view here before we start fleshing out actual JIRA tasks to >>> add to https://issues.apache.org/jira/browse/SOLR-14168 >>> <https://issues.apache.org/jira/browse/SOLR-14168>. >>> >>> If we end up releasing official Solr Docker images together with the normal >>> release, it would be cool to add documentation to the RefGuide and perhaps >>> tutorial, on how to run Solr with Docker. >>> >>> Jan >>> >>> >> >
