Hi Vedarth, I think we shouldn't introduce any new release process that is docker specific. We should consider the software in the docker image in the same way as consider third party dependencies today - if there is a high severity CVE affecting any of them, we aim to release a new version of Kafka for the affected branch. It would include the latest Kafka code from the branch.
Additionally, we should specify the EOL policy in this KIP - we are not changing it as part of it. One interesting detail is that the release document claims we support the last 3 releases, but the reality has been a bit different - we tend to support the 2 most recent releases unless it's a high severity CVE in Kafka itself (these tend to be much rarer, thankfully). Ismael On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma <vedarth.sha...@gmail.com> wrote: > Hi Mickael, > Thanks for going through the KIP and providing valuable feedback. > > 1. We will support the latest LTS version of Java supported by Apache > Kafka. > 2. We will provide support for the last three releases. We've added a > detailed example of this in the KIP under our EOL policy. > 3. We can establish a nightly cron job using GitHub Actions and leverage > an open-source vulnerability scanning tool like trivy ( > https://github.com/aquasecurity/trivy), to get vulnerability reports on > all supported images. This tool offers a straightforward way to > integrate > vulnerability checks directly into our GitHub Actions workflow. > 4. That's a good suggestion to have a GitHub Actions workflow. We will > implement a GitHub Actions workflow to automate the build and testing > process. > 5. Regarding the release process, we observed that there isn't an > existing CI/CD pipeline. We can consider the addition of a GitHub > workflow > to facilitate the release process. > > Please let us know your thoughts on the above. > > Thanks and regards, > Vedarth > > On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison <mickael.mai...@gmail.com> > wrote: > > > Hi Krishna, > > > > Overall I'm supportive of having an official docker image. > > I have a few questions: > > - Can you clarify the process of selecting the Java version? Is the > > proposal to only pick LTS versions? or to pick the highest version > > supported by Kafka? > > - Once a new Kafka version is released, what happens to the image > > containing the previous release? Do we expect to still update it in > > case of CVEs? If so for how long? > > - How will we get notified that the base image has a CVE? > > - Rather than having scripts PMC members have to run from their > > machines, would it e possible to have a Jenkins job or GitHub action? > > > > Thanks, > > Mickael > > > > > > > > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma > > <vedarth.sha...@gmail.com> wrote: > > > > > > Hi Manikumar, > > > > > > Thanks for the feedback! > > > > > > 1. We propose the addition of a new directory named "docker" at the > root > > of > > > the repository, where all Docker-related code will be stored. A > detailed > > > directory structure has been added in the KIP. > > > 2. We request the creation of an Apache Kafka repository (apache/kafka) > > on > > > DockerHub, to be administered under the The Apache Software Foundation > > > <https://hub.docker.com/u/apache>. The PMC members should have the > > > necessary permissions for pushing updates to the docker repo. > > > > > > Thanks and regards, > > > Vedarth > > > > > > > > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > > > Hi Krishna, Vedarth, > > > > > > > > Thanks for the KIP. > > > > > > > > 1. Can we add directory structure of Docker Image related files in > > Kafka > > > > repo. > > > > > > > > 2. > Steps for the Docker image release will be included in the > Release > > > > Process doc of Apache Kafka > > > > > > > > Can we list down the requirements (repos, accounts) for releasing > > images to > > > > docker hub. I am mainly asking because PMC needs to request docker > hub > > > > access/repos. > > > > I can help in getting required repos/accounts. > > > > https://infra.apache.org/docker-hub-policy.html > > > > > > > > > > > > Thanks, > > > > Manikumar > > > > > > > > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal < > > > > krishna0608agar...@gmail.com> wrote: > > > > > > > > > Hi Viktor, > > > > > > > > > > I've noticed there are two types of custom jar configurations: > > > > > > > > > > 1. *Type 1*: In this case, only the class name is required(e.g > > > > > *authorizer.class.name > > > > > <http://authorizer.class.name>**)* This can be configured by > the > > > > > following steps: > > > > > - Mount the jar in the container. > > > > > - Configure the *CLASSPATH* environment variable (used by > > > > > *kafka-run-class.sh*) by providing the mounted path to it. > > This can > > > > > be passed as an environment variable to the docker container. > > > > > 2. *Type 2*: Here, in addition to the class name, classpath can > > also > > > > be > > > > > configured (eg *remote.log.metadata.manager.class.name > > > > > <http://remote.log.metadata.manager.class.name> *and > > > > > *remote.log.metadata.manager.class.path*). This can be > configured > > by > > > > the > > > > > following steps: > > > > > - Mount the jar in the container. > > > > > - Configure the respective *class.path* property. > > > > > > > > > > Regards, > > > > > Krishna > > > > > > > > > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal < > > > > > krishna0608agar...@gmail.com> wrote: > > > > > > > > > > > Hi Viktor, > > > > > > Thanks for the questions. > > > > > > > > > > > > 1. While the docker image outlined in KIP-975 is designed for > > > > > > production environments, it is equally suitable for > development > > and > > > > > testing > > > > > > purposes. We will furnish the docker image, allowing users the > > > > > flexibility > > > > > > to employ it according to their specific needs. > > > > > > 2. The configs will be injected into the docker container > > through > > > > > > environment variables. These environment variables will have a > > > > prefix > > > > > > allowing for efficient parsing to extract the relevant > > > > > properties.(Will add > > > > > > this implementation in the KIP as well once we converge on > > this.) > > > > > > 3. Regarding this question, I'll conduct a test on my end > after > > > > > > gaining a better understanding, and then provide you with a > > > > response. > > > > > > > > > > > > Regards, > > > > > > Krishna > > > > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass > > > > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > > > > > >> Hi Ismael, > > > > > >> > > > > > >> I'm not trying to advocate against the docker image, I just > > pointed > > > > out > > > > > >> that the current scoping of the KIP may be a bit too generic and > > > > thought > > > > > >> that KIP-974 and KIP-975 were aiming for mostly the same thing > > and can > > > > > be > > > > > >> discussed under one umbrella. Apologies if this was rooted in a > > > > > >> misunderstanding. > > > > > >> > > > > > >> Kirshna, > > > > > >> > > > > > >> I think we need to refine the KIP a bit more. I think there are > > some > > > > > >> interfaces that we need to include in the KIP as Kafka has > > plugins in > > > > > >> certain cases where users are expected to provide implementation > > and I > > > > > >> think it's worth discussing this in the KIP as they're kind of > > > > > interfaces > > > > > >> for users. Here are my questions in order: > > > > > >> 1. In what environments do you want the image to be used? As I > > > > > understand > > > > > >> it would replace the current testing image and serve as a basis > > for > > > > > >> development, but would it aim at production use cases too > > > > > (docker-compose, > > > > > >> Kubernetes, etc.)? > > > > > >> 2. How do you plan to forward configs to the broker? Do we > expect > > a > > > > > >> populated server.properties file placed in a certain location or > > > > should > > > > > >> the > > > > > >> docker image create this file based on some input (like env > vars)? > > > > > >> 3. Certain parts can be pluggable, like metric reporters or > > remote log > > > > > >> implementations that were just introduced by KIP-405. These > > manifest > > > > in > > > > > >> jar > > > > > >> files that must be put on the classpath of Kafka while certain > > > > > classnames > > > > > >> have to be configured. How do you plan to implement this, how do > > we > > > > > >> allow users to configure such things? > > > > > >> > > > > > >> Thanks, > > > > > >> Viktor > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole > > > > > >> <kevers...@cloudflare.com.invalid> wrote: > > > > > >> > > > > > >> > Hello, > > > > > >> > > > > > > >> > I think this would be a wonderful improvement to the > ecosystem. > > > > While > > > > > >> > Viktor is correct that most Docker pipelines eventually lead > to > > a > > > > > >> > kubernetes deployment, that should not stop us from creating > an > > > > > >> > Official Docker Image. Creating a Docker image would allow us > to > > > > > ensure > > > > > >> a > > > > > >> > level of quality and support for people who want to deploy > > Kafka as > > > > a > > > > > >> > container on baremetal machines, it could allow us to create > > > > > >> > a sandbox/developer environment for new contributors and > > developers > > > > to > > > > > >> test > > > > > >> > and have a single agreed upon environment that kafka works in > > for > > > > > future > > > > > >> > KIPs and would most likely spawn more contributions from > people > > > > > wanting > > > > > >> to > > > > > >> > optimize kafka for k8s. > > > > > >> > > > > > > >> > > > > > > >> > I am 100% for this and will gladly help if approved. > > > > > >> > > > > > > >> > Kenneth > > > > > >> > > > > > > >> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma < > m...@ismaeljuma.com> > > > > > wrote: > > > > > >> > > > > > > >> > > Hi Viktor, > > > > > >> > > > > > > > >> > > I disagree. Docker is a very popular deployment tool and > it's > > not > > > > > only > > > > > >> > used > > > > > >> > > with Kubernetes. > > > > > >> > > > > > > > >> > > Ismael > > > > > >> > > > > > > > >> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass > > > > > >> > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > >> > > > > > > > >> > > > Hi Krishna, > > > > > >> > > > > > > > > >> > > > I think you should merge this KIP and KIP-974 > > > > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974> > as > > > > there > > > > > >> are > > > > > >> > overlaps as > > > > > >> > > > Federico pointed out on KIP-974 > > > > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974>. > I > > > > think > > > > > >> you > > > > > >> > should keep that one as it > > > > > >> > > > has well defined goals (improve tests) while I feel this > > one is > > > > > too > > > > > >> > > > generic. Docker is usually just a tool for either testing > or > > > > > >> > Kubernetes, > > > > > >> > > so > > > > > >> > > > they have very well defined use-cases. In the case of > Flink > > for > > > > > >> > instance > > > > > >> > > > the image is used for its kubernetes operator. The use > case > > > > would > > > > > >> > > determine > > > > > >> > > > a lot of things and I think a generic image would likely > > not fit > > > > > the > > > > > >> > > needs > > > > > >> > > > of all use-cases. > > > > > >> > > > > > > > > >> > > > Best, > > > > > >> > > > Viktor > > > > > >> > > > > > > > > >> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal < > > > > > >> > > > krishna0608agar...@gmail.com> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > >> > > > > Hi, > > > > > >> > > > > Apache Kafka does not have an official docker image > > currently. > > > > > >> > > > > I want to submit a KIP to publish a docker image for > > Apache > > > > > Kafka. > > > > > >> > > > > > > > > > >> > > > > KIP-975 < > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975>: > > > > > >> > Docker Image for Apache Kafka > > > > > >> > > > > < > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > Regards, > > > > > >> > > > > Krishna > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >
