Hi Krish and Prabha, Thanks for your replies. I still have some follow-up questions:
1) I read https://docs.docker.com/trusted-content/official-images/ and didn't find anything on that page or immediately around it that explains what compliance requirements might be satisfied by a DOI that couldn't be satisfied by the existing apache/kafka image. Can you elaborate? Feel free to provide another link, but please also quote the relevant sections from it (as StackOverflow likes to say, links can grow stale over time). 2) It would be great to see a brief summary of the differences in these images included in the KIP, in order to try to gauge how this would look to our users. 3) What I suggested last time was not a separate apache/apache-docker repository, but a repository controlled entirely by Docker. The DOI docs [1] state that "While it's preferable to have upstream software authors maintaining their Docker Official Images, this isn't a strict requirement", which I take to mean that it's not required for an Apache Kafka DOI to live under the apache organization on GitHub. It also seems like there's precedent for this: images for MySQL [2] and PHP [3] already exist under the control of Docker. The reason I think this is worth considering is that Docker can arbitrarily change the eligibility requirements for their official images at any time, and it doesn't seem like there's a clear process in the KIP for judging how we should respond to these changes (in fact, it seems like the idea in the KIP is that we should make any change required with no further vetting beyond possibly a pull request on apache/kafka, which would require approval by a committer). By hosting the DOI definitions ourselves (either in apache/kafka, or in a theoretical apache/docker-kafka repository), we take responsibility for the image, even if the owner on Docker Hub is Docker, not Apache. If the code lives elsewhere, then (as long as basic trademark and possibly security guidelines are respected) Apache doesn't have to concern itself at all with the image and the maintainers are free to make whatever changes they want to it in order to meet Docker's requirements. [1] - https://docs.docker.com/trusted-content/official-images/contributing/, second paragraph under "Contributing to Docker Official Images" [2] - https://github.com/docker-library/mysql [3] - https://github.com/docker-library/php Cheers, Chris On Fri, May 10, 2024 at 7:46 AM Krish Vora <krishvor...@gmail.com> wrote: > Hey Chris, > > We have responded to the initial round of queries. And as you mentioned in > your comment, you may have more questions related to this KIP. Please let > us know if you have any. > We want to start the voting for this KIP soon, as we intend to include it > in the 3.8.0 release. > > Thanks. > > Regards, > > Krish. > > On Wed, May 8, 2024 at 6:19 PM Krish Vora <krishvor...@gmail.com> wrote: > > > Hi Chris. Thanks for the questions. > > > > 3. Would a separate Docker-owned repository be out of the question? I'm > >> guessing there are some trademark issues that might get in the way, but > >> it's worth exploring since the entire purpose of this KIP seems to be to > >> provide images that are vetted and designed by Docker more than by the > >> Apache Kafka contributors/committers/PMC. > > > > > > > > - The process for introducing a Docker Official Image involves > > - Hosting the Dockerfile in the Apache Kafka repository and > > - Providing the path to this Dockerfile to Docker Hub in Docker > > Hub’s own repo > > < > https://github.com/docker-library/official-images/tree/master/library> > > . > > - This ensures that any updates to the Dockerfile in the AK repository > > are directly applicable to the docker official images available on > Docker > > Hub. > > > > > > - We also did not find any added advantage to create a separate > > repository named apache-docker within the Apache GitHub organization. > > > > Thanks, > > Krish. > > > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli > > <mpra...@confluent.io.invalid> wrote: > > > >> Hi Chris, I would like to add more context to this KIP's motivation. > >> Vedarth and Krish, please weigh in with your inputs. > >> > >> In the motivation section it's stated that "Several other Apache > projects, > >> > like Flink, Spark, Solr, have already released Docker Official Images, > >> with > >> > download figures ranging from 50 million to over 1 billion. These > >> numbers > >> > highlight the significant demand among users." But then immediately > >> > afterwards, we learn that "Also the Docker Official Images are always > >> the > >> > top 1 search result, irrespective of the number of downloads." > Wouldn't > >> a > >> > high number of downloads for an image naturally follow from being the > >> top > >> > search result? It seems like we can't necessarily assume that Docker > >> > Official Images are inherently more desirable for users based solely > on > >> > download statistics. > >> > > >> > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official image > >> is > >> more desirable for workloads that have stringent compliance > requirements. > >> More details on why official images are more trusted are documented here > >> <https://docs.docker.com/trusted-content/official-images/>. The Docker > >> Official image would also help an absolutely new Kafka beginner who > might > >> not know about Apache or the concept of Sponsored images. We want to > make > >> it easier for Kafka beginners to discover the Kafka image through > >> DockerHub. > >> > >> > >> Can you elaborate on the value that these new images would add from a > >> > user's perspective? I'm hesitant to introduce another image, since it > >> adds > >> > to the cognitive burden of people who will inevitably have to answer > the > >> > question of "What are the differences between all of the available > >> images > >> > and which one is best for my use case?" > >> > > >> > >> > >> *My thoughts: *This is a valid concern to address. The response to the > >> above question addresses the value-add this new Docker Official image > >> would > >> provide. I also agree we need a clear distinction between each of these > >> images to be well documented. We plan to update the AK website with > >> details > >> on how, why, and when a developer would want to use each of these > >> particular images(KIP-974,975,1028). > >> > >> Thanks, > >> Prabha. > >> > >> > >> > >> > >> > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton <chr...@aiven.io.invalid> > >> wrote: > >> > >> > Hi Vedarth and Krish, > >> > > >> > Thanks for the KIP! I have to admit I'm a little skeptical; hopefully > >> you > >> > can help me understand the need for these additional images. > >> > > >> > 1) In the motivation section it's stated that "Several other Apache > >> > projects, like Flink, Spark, Solr, have already released Docker > Official > >> > Images, with download figures ranging from 50 million to over 1 > billion. > >> > These numbers highlight the significant demand among users." But then > >> > immediately afterwards, we learn that "Also the Docker Official Images > >> are > >> > always the top 1 search result, irrespective of the number of > >> downloads." > >> > Wouldn't a high number of downloads for an image naturally follow from > >> > being the top search result? It seems like we can't necessarily assume > >> that > >> > Docker Official Images are inherently more desirable for users based > >> solely > >> > on download statistics. > >> > > >> > 2) Can you elaborate on the value that these new images would add > from a > >> > user's perspective? I'm hesitant to introduce another image, since it > >> adds > >> > to the cognitive burden of people who will inevitably have to answer > the > >> > question of "What are the differences between all of the available > >> images > >> > and which one is best for my use case?" > >> > > >> > 3) Would a separate Docker-owned repository be out of the question? > I'm > >> > guessing there are some trademark issues that might get in the way, > but > >> > it's worth exploring since the entire purpose of this KIP seems to be > to > >> > provide images that are vetted and designed by Docker more than by the > >> > Apache Kafka contributors/committers/PMC. > >> > > >> > I may have more questions later but wanted to get this initial round > out > >> > now without trying to list everything first. > >> > > >> > Looking forward to your thoughts! > >> > > >> > Cheers, > >> > > >> > Chris > >> > > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma < > >> vedarth.sha...@gmail.com> > >> > wrote: > >> > > >> > > Hey folks, > >> > > > >> > > Thanks a lot for reviewing the KIP and providing feedback. > >> > > The discussion thread seems resolved and KIP has been updated > >> > accordingly. > >> > > We will be starting the voting thread for this KIP in the next few > >> days. > >> > > Please take a look at the KIP and let us know if any further > >> discussion > >> > > is needed. > >> > > > >> > > Thanks and regards, > >> > > Vedarth > >> > > > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar < > manikumar.re...@gmail.com> > >> > > wrote: > >> > > > >> > > > Thanks Krish. KIP looks good to me. > >> > > > > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <krishvor...@gmail.com > > > >> > > wrote: > >> > > > > > >> > > > > Hi Manikumar, > >> > > > > > >> > > > > Thanks for the comments. > >> > > > > > >> > > > > Maybe as part of the release process, RM can create a JIRA for > >> this > >> > > > > > task. This can be taken by RM or any comitter or any > contributor > >> > > (with > >> > > > > > some help from commiters to run "Docker Image Preparation via > >> > GitHub > >> > > > > > Actions:" > >> > > > > > >> > > > > This sounds like a good idea. This step would be beneficial. By > >> > > creating > >> > > > a > >> > > > > JIRA ticket, it will also serve as a reminder to complete the > >> > > > post-release > >> > > > > steps for the Docker official images. Have updated the KIP with > >> this > >> > > > step. > >> > > > > > >> > > > > Is this using GitHub Actions workflow? or manual testing? > >> > > > > > >> > > > > This will be done by a Github Actions workflow, which will test > >> the > >> > > > static > >> > > > > Docker Official Image assets for a specific release version. > >> > > > > > >> > > > > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s > >> > > > > > official images repository (or) can it be done by any > >> contributor. > >> > > > > > >> > > > > I believe that it can be done by any contributor (ref: This link > >> > > > > < > >> > https://docs.docker.com/trusted-content/official-images/contributing/ > >> > > > > >> > > > > quotes "*Anyone can provide feedback, contribute code, suggest > >> > process > >> > > > > changes, or even propose a new Official Image.*") > >> > > > > > >> > > > > Also I was thinking, once the KIP gets voted, we should try to > >> > release > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help > us > >> to > >> > > > > > validate the process and allow us to fix any changes suggested > >> by > >> > > > > > Dockerhub before the 3.8.0 release. > >> > > > > > >> > > > > This sounds like a great idea. This KIP proposes release of DOI > >> as a > >> > > > > post-release process, which can be done anytime post release. > >> Since > >> > > 3.7.0 > >> > > > > is already released, we can perform these steps for that release > >> too. > >> > > By > >> > > > > the time the KIP gets implemented, if 3.7.1 is released, we > could > >> do > >> > > > these > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to make > >> > changes > >> > > to > >> > > > > the Dockerfiles and other assets based on feedback from Docker > Hub > >> > > before > >> > > > > the release of version 3.8.0. > >> > > > > > >> > > > > Thanks, > >> > > > > Krish. > >> > > > > > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar < > >> > manikumar.re...@gmail.com> > >> > > > > wrote: > >> > > > > > >> > > > > > Hi Krish, > >> > > > > > > >> > > > > > Thanks for the updated KIP. a few comments below. > >> > > > > > > >> > > > > > > "These actions can be carried out by the RM or any > contributor > >> > post > >> > > > the > >> > > > > > release process." > >> > > > > > Maybe as part of the release process, RM can create a JIRA for > >> this > >> > > > > > task. This can be taken by RM or any comitter or any > contributor > >> > > (with > >> > > > > > some help from commiters to run "Docker Image Preparation via > >> > GitHub > >> > > > > > Actions:" > >> > > > > > > >> > > > > > > "Perform Docker build tests to ensure image integrity" > >> > > > > > Is this using GitHub Actions workflow? or manual testing? > >> > > > > > > >> > > > > > > "The RM will manually raise the final PR to Docker Hub’s > >> official > >> > > > images > >> > > > > > repository using the contents of the generated file" > >> > > > > > Is it mandatory for RM/comitters to raise the PR to Docker > >> Hub’s > >> > > > > > official images repository (or) can it be done by any > >> contributor. > >> > > > > > > >> > > > > > Also I was thinking, once the KIP gets voted, we should try to > >> > > release > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help > us > >> to > >> > > > > > validate the process and allow us to fix any changes suggested > >> by > >> > > > > > Dockerhub before the 3.8.0 release. > >> > > > > > > >> > > > > > > >> > > > > > Thanks, > >> > > > > > > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora < > >> krishvor...@gmail.com> > >> > > > wrote: > >> > > > > > > > >> > > > > > > Hi Manikumar and Luke. > >> > > > > > > Thanks for the questions. > >> > > > > > > > >> > > > > > > 1. No, the Docker inventory files and configurations will > not > >> be > >> > > the > >> > > > same > >> > > > > > > for Open Source Software (OSS) Images and Docker Official > >> Images > >> > > > (DOI). > >> > > > > > > > >> > > > > > > For OSS images, the Dockerfile located in > >> docker/jvm/dockerfile > >> > is > >> > > > > > > utilized. This process is integrated with the existing > release > >> > > > pipeline > >> > > > > > as > >> > > > > > > outlined in KIP-975 > >> > > > > > > < > >> > > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status > >> > > > > > >, > >> > > > > > > where the Kafka URL is provided as a build argument. This > >> method > >> > > > allows > >> > > > > > for > >> > > > > > > building, testing, and releasing OSS images dynamically. The > >> OSS > >> > > > images > >> > > > > > > will continue to be released under the standard release > >> process . > >> > > > > > > > >> > > > > > > In contrast, the release process for DOIs requires providing > >> the > >> > > > Docker > >> > > > > > Hub > >> > > > > > > team with a specific directory for each version release that > >> > > > contains a > >> > > > > > > standalone Dockerfile. These Dockerfiles are designed to be > >> > > > > > > self-sufficient, hence require hardcoded values instead of > >> > relying > >> > > on > >> > > > > > build > >> > > > > > > arguments. To accommodate this, in our proposed approach, a > >> new > >> > > > directory > >> > > > > > > named docker_official_images has been created. This > directory > >> > > > contains > >> > > > > > > version-specific directories, having Dockerfiles with > >> hardcoded > >> > > > > > > configurations for each release, acting as the source of > truth > >> > for > >> > > > DOI > >> > > > > > > releases. The hardcoded dockerfiles will be created using > the > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of post > >> > release > >> > > we > >> > > > > > will > >> > > > > > > be creating a Dockerfile that will be reviewed by the > >> Dockerhub > >> > > > community > >> > > > > > > and might need changes as per their review. This approach > >> ensures > >> > > > that > >> > > > > > DOIs > >> > > > > > > are built consistently and meet the specific requirements > set > >> by > >> > > > Docker > >> > > > > > Hub. > >> > > > > > > > >> > > > > > > 2. Yes Manikumar, transitioning the release of Docker > Official > >> > > Images > >> > > > > > (DOI) > >> > > > > > > to a post-release activity does address the concerns about > >> > > > complicating > >> > > > > > the > >> > > > > > > release process. Initially, we considered incorporating DOI > >> > release > >> > > > > > > directly into Kafka's release workflow. However, this > approach > >> > > > > > > significantly increased the RMs workload due to the addition > >> of > >> > > > numerous > >> > > > > > > steps, complicating the process. By designating the DOI > >> release > >> > as > >> > > a > >> > > > > > > post-release task, we maintain the original release process. > >> This > >> > > > > > > adjustment allows for the DOI release to be done after the > >> main > >> > > > release. > >> > > > > > We > >> > > > > > > have revised the KIP to reflect that DOI releases will now > >> occur > >> > > > after > >> > > > > > the > >> > > > > > > main release phase. Please review the updated document and > >> > provide > >> > > > any > >> > > > > > > feedback you might have. > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > Krish. > >> > > > > > > > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <show...@gmail.com > > > >> > > wrote: > >> > > > > > > > >> > > > > > > > Hi Krishna, > >> > > > > > > > > >> > > > > > > > I also have the same question as Manikumar raised: > >> > > > > > > > 1. Will the Docker inventory files/etc are the same for > OSS > >> > Image > >> > > > and > >> > > > > > > > Docker Official Images? > >> > > > > > > > If no, then why not? Could we make them identical so that > we > >> > > don't > >> > > > > > have to > >> > > > > > > > build 2 images for each release? > >> > > > > > > > > >> > > > > > > > Thank you. > >> > > > > > > > Luke > >> > > > > > > > > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar < > >> > > > manikumar.re...@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi Krishna, > >> > > > > > > > > > >> > > > > > > > > Thanks for the KIP. > >> > > > > > > > > > >> > > > > > > > > I think Docker Official Images will be beneficial to the > >> > Kafka > >> > > > > > community. > >> > > > > > > > > Few queries below. > >> > > > > > > > > > >> > > > > > > > > 1. Will the Docker inventory files/etc are the same for > >> OSS > >> > > > Image and > >> > > > > > > > > Docker Official Images > >> > > > > > > > > 2. I am a bit worried about the new steps to the release > >> > > process. > >> > > > > > Maybe > >> > > > > > > > we > >> > > > > > > > > should consider Docker Official Images release as > >> > Post-Release > >> > > > > > activity. > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora < > >> > > > krishvor...@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Hi Hector, > >> > > > > > > > > > > >> > > > > > > > > > Thanks for reaching out. This KIP builds on top of > >> KIP-975 > >> > > > > > > > > > < > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka > >> > > > > > > > > > > > >> > > > > > > > > > and > >> > > > > > > > > > aims to introduce a JVM-based Docker Official Image > (DOI > >> > > > > > > > > > < > >> https://docs.docker.com/trusted-content/official-images/ > >> > >) > >> > > > for > >> > > > > > Apache > >> > > > > > > > > > Kafka that will be visible under Docker Official > Images > >> > > > > > > > > > < > https://hub.docker.com/search?image_filter=official&q= > >> >. > >> > > Once > >> > > > > > > > > implemented > >> > > > > > > > > > for Apache Kafka, for each release, there will be one > >> more > >> > > > > > JVM-based > >> > > > > > > > > Docker > >> > > > > > > > > > image available to users. > >> > > > > > > > > > > >> > > > > > > > > > Currently, we already have an OSS sponsored image, > which > >> > was > >> > > > > > introduced > >> > > > > > > > > via > >> > > > > > > > > > KIP-975 (apache/kafka < > >> > > > https://hub.docker.com/r/apache/kafka/tags > >> > > > > > >) > >> > > > > > > > > which > >> > > > > > > > > > comes under The Apache Software Foundation < > >> > > > > > > > > > https://hub.docker.com/u/apache> in > >> > > > > > > > > > Docker Hub. The new Docker Image is the Docker > Official > >> > Image > >> > > > > > (DOI), > >> > > > > > > > > which > >> > > > > > > > > > will be built and maintained by Docker Community. > >> > > > > > > > > > > >> > > > > > > > > > For example, for a release version like 3.8.0 we will > >> have > >> > > two > >> > > > JVM > >> > > > > > > > based > >> > > > > > > > > > docker images:- > >> > > > > > > > > > > >> > > > > > > > > > - apache/kafka:3.8.0 (OSS sponsored image) > >> > > > > > > > > > - kafka:3.8.0 (Docker Official image) > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > I have added the same in the KIP too for everyone's > >> > > reference. > >> > > > > > > > > > Thanks, > >> > > > > > > > > > Krish. > >> > > > > > > > > > > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino > >> > (BLOOMBERG/ > >> > > > 919 > >> > > > > > 3RD > >> > > > > > > > A) < > >> > > > > > > > > > hgerald...@bloomberg.net> wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Hi, > >> > > > > > > > > > > > >> > > > > > > > > > > What is the difference between this KIP and KIP-975: > >> > Docker > >> > > > > > Image for > >> > > > > > > > > > > Apache Kafka? > >> > > > > > > > > > > > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 > >> > > UTC-4:00To: > >> > > > > > > > > > > dev@kafka.apache.org > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image > for > >> > > Apache > >> > > > > > Kafka > >> > > > > > > > > > > > >> > > > > > > > > > > Hi everyone, > >> > > > > > > > > > > > >> > > > > > > > > > > I would like to start the discussion on > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im > >> > > > > > > > > > > age+for+Apache+Kafka > >> > > > > > > > > > > . > >> > > > > > > > > > > > >> > > > > > > > > > > This KIP aims to introduce JVM based Docker Official > >> > Image > >> > > > (DOI) > >> > > > > > for > >> > > > > > > > > > Apache > >> > > > > > > > > > > Kafka. > >> > > > > > > > > > > > >> > > > > > > > > > > Regards, > >> > > > > > > > > > > Krish. > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > >> > > >> > >> > >> -- > >> > >> [image: Confluent] <https://www.confluent.io> > >> Prabha Manepalli > >> Product Manager for Confluent Platform Security, Docker > >> linkedin.com/in/prabhamanepalli > >> > > >
