Thanks for bringing up this issue Kevin. Periodically re-building docker images is certainly a feature we could support, and probably should to automatically keep up with security fixes. We could even automate it pretty easily in Jenkins.
We could also build in support in the gradle commands to instead of building a TGZ from source, download and verify the "official" TGZ, to build the image with. That way release images are always built with the same exact binaries. The Dockerfile wouldn't need to change at all between local and release, it still merely expects a TGZ to be passed in the context; gradle can determine if it needs to be built from scratch or downloaded. This still likely wouldn't be good enough to make the image an "official docker image" but it gets us to essentially the same end-state image. The only difference is the downloading and verification are happening in gradle instead of the Dockerfile. - Houston On Fri, Jan 15, 2021 at 2:05 PM Kevin Risden <kris...@apache.org> wrote: > Currently the solr-docker-image, and a majority of "Docker Official >> Images", the officially released Solr binaries are downloaded from mirrors >> and validated within the Dockerfiles. This makes it easy to ensure to users >> that the 9.0 solr docker image contains the 9.0 solr release. This process >> doesn't fit very well with local builds, because there is nowhere to >> download local builds from, and validation isn't required. >> >> *The current opinion in the community is to abandon the "Docker Official >> Images" style process of downloading and validating official binaries, and >> instead having the release manager use the local-build image creation with >> the final release source.* This should result in the same docker image >> in the end, however there is no trust built into the docker image itself. >> Instead we are likely going to document a way for users to verify the >> docker-image contents themselves. >> > > Before we abandon the official process of downloading/validating official > binaries, I think there is a good reason to keep the ability to download an > "official" Apache Solr release and use it in the "official" Solr > convenience Docker image. > > Docker images are static point in time copies of an OS and all supporting > packages (like Java) when built. Periodically Docker images should be > rebuilt to pick up the latest security and bug enhancements in the base > image. Just like any OS should run `apt upgrade` or `yum update` > periodically to ensure it is up to date. > > My proposal is to periodically rebuild the "official" Solr convenience > Docker image based on the "official" Solr release to ensure we keep the > Docker images up to date. The idea being that we have a list of "supported" > versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly) > the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets > added to this rebuilding matrix. This ensures that the "official" Solr > convenience Docker image is reasonably up to date with regards to the base > image security updates. > > My understanding (from a few years ago) is that the Docker official images > are rebuilt when the base image is updated. This was automatic from what I > understand. If we move away from the "Docker official image" way of doing > things, we should still ensure that we can provide high quality secure > Docker images to the community. > > We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update > the Docker image to pick up the latest base image changes. > > It is important to be able to build and test locally with a Docker image > that matches what an "official" Docker image is. This should still be a > goal, but we should be able to rebuild the Solr docker image without > rebuilding all of Solr. > > PS - I chatted a bit with Houston on Slack about this topic and hopefully > captured all the context correctly. > > Kevin Risden > > > On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <houstonput...@gmail.com> > wrote: > >> There's a few decisions that need to be ironed out around the Solr docker >> image before 9.0 is released. This is because the community has decided >> that Solr should start releasing it's own docker images starting with 9.0. >> >> Below is the current state of the ongoing discussions for the Solr Docker >> image. Please feel free to correct me or fill in any information I may be >> missing. >> >> Where does this image live? >> >> There are two options for this really. >> >> - _/solr - docker run solr:9.0 (Official Docker Image) >> - apache/solr - docker run apache/solr:9.0 >> >> The benefits of the first are 1) the nice usability of being able to >> plainly specify "solr" and 2) the "Docker Official Images" badge on >> DockerHub. The downsides are that there are very strict requirements with >> creating Official Docker Images, which would complicate and require >> separating the way that we build release docker images and local docker >> images. >> >> The benefits of using the apache namespace is that we can build the image >> in any way that we want. We would be able to build release and local docker >> images the exact same way. The downside is the loss of the "Docker Official >> Images" badge. >> >> *I think there is some consensus that choosing the "apache/solr" location >> is fine, and worth the added flexibility we get in the build process.* >> >> Legal Stuff >> >> There are a few legal questions we need to keep in mind when creating >> this process and doing a release for the first time. >> >> - Source release - The apache policy is: (from Michael Sokolov) >> >> “Every ASF release MUST contain one or more source packages, which >>> MUST be sufficient for a user to build and test the release provided they >>> have access to the appropriate platform and tools.” >> >> For the docker build this is fine as long as the solr/docker gradle >> module is included in the source release. As one can always rebuild the >> same image running gradlew docker using the source. >> >> - >> >> Jan Høydahl mentioned that the docker file layers should be limited, >> but I'm not exactly sure what this means or entails. Maybe he can expand >> on >> this. >> >> Artifacts within the Image >> >> As mentioned above in the "Image Location" section, the goal of including >> the docker build process inside the Solr project is to make development >> easier by providing an easy way to build the official-style docker image >> with local source code. In order to achieve "official-style" images for >> local builds, we want to make the build process for local images as close >> as possible to the process for building official release images. >> >> Currently the solr-docker-image, and a majority of "Docker Official >> Images", the officially released Solr binaries are downloaded from mirrors >> and validated within the Dockerfiles. This makes it easy to ensure to users >> that the 9.0 solr docker image contains the 9.0 solr release. This process >> doesn't fit very well with local builds, because there is nowhere to >> download local builds from, and validation isn't required. >> >> *The current opinion in the community is to abandon the "Docker Official >> Images" style process of downloading and validating official binaries, and >> instead having the release manager use the local-build image creation with >> the final release source.* This should result in the same docker image >> in the end, however there is no trust built into the docker image itself. >> Instead we are likely going to document a way for users to verify the >> docker-image contents themselves. >> >> I am not sure what the user-side verification process would look like for >> the image, but I definitely think it is something that we should look into. >> Maybe Docker will allow us to use these as official images if we can script >> out this verification and make it easy for them to do? Just a thought, I'm >> not sure if that would actually work. >> >
