Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hey all, As the release manager for 3.7.0, I am pretty interested to know if we should consider this a blocker. Do we have clarity as to whether users could practically rely on this Go script? From a shallow look, it's only used in one line in the Dockerfile. I guess the downside is that images

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

We should be very clear on what users can rely on when it comes to the docker images (i.e. what are public interfaces) and what are implementation details (and can be changed whenever we want). That's the only way to have a maintainable system. Same way we make changes to internal classes even

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi, Yes changes have to be merged by a committer but for this kind of decisions it's best if it's seen by more than one. > Hmm, is this a blocker? I don't see why. It would be nice to include it in > 3.7 and we have time, so I'm fine with that. Sure, it's not a blocker in the usual sense. But

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Mickael, A couple of comments inline. On Wed, Dec 20, 2023 at 3:34 AM Mickael Maison wrote: > When you say, "we have opted to take a different approach", who is > "we"? I think this decision should be made by the committers. > Changes can only be merged by committers, so I think it's

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Vedarth, Thanks for the quick reply and for the clarifications. I agree with your analysis and the alternative approach you found makes sense. I'll try to take a look at your PR later on today or tomorrow. Thanks, Mickael On Wed, Dec 20, 2023 at 1:33 PM Vedarth Sharma wrote: > > Hi

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Mickael, In our previous approach startup time would have been affected if the utility script code was written in some other language. If we had just replaced the utility script with java code that would have impacted startup time significantly. In my local machine, it took around 50ms (on M2

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Vedarth, In the PR when I raised concerns about introducing Go to the repository you responded "Startup time will get impacted though, as java is significantly slower than golang and we are calling this script multiple times.". It would be good to provide numbers if you have, so the community

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hello everyone, In the course of implementing this KIP, we introduced a small piece of code in golang to prepare property files for the Docker image. Our decision was influenced by considerations such as performance, code testability, and a reduction in the final Docker image size, as outlined in

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Vedarth, > Local Kafka startup time (without JSA): 1.592 secs > Local Kafka startup time (with JSA): 1.016 secs > Local Kafka startup memory usage (without JSA): 440MB > Local Kafka startup memory usage (with JSA): 380MB This is a significant reduction in start-up time (33%) - nice! Ismael

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Ismael! Thanks for bringing this to our attention. We did a small POC integrating CDS with Kafka server startup, and encountered positive outcomes(results are added in the KIP). Hence, we've decided to include the dynamically generated JSA file from the following workflow in the Docker image:

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

The reference I meant to include: https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html On Tue, Oct 24, 2023, 10:25 PM Ismael Juma wrote: > Hi Krishna, > > One last question from me, did we confuse using AppCDS or Dynamic CDS? > > Thanks, > Ismael > > On Tue, Oct 24, 2023, 9:54 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, One last question from me, did we confuse using AppCDS or Dynamic CDS? Thanks, Ismael On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal wrote: > Hi, > Thanks for the insightful feedback on this KIP. If there are no further > questions, I'm considering wrapping up this discussion

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi, Thanks for the insightful feedback on this KIP. If there are no further questions, I'm considering wrapping up this discussion thread. We'll be moving into the voting process in the next couple of days. Your continued input is greatly appreciated! Regards, Krishna On Fri, Sep 8, 2023 at 1:27 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Ismael, Thanks for the correction. We will be following the same EOL policy as Apache Kafka, hence we have removed it from the KIP. Thanks and regards, Vedarth On Mon, Oct 23, 2023 at 12:17 PM Ismael Juma wrote: > Sorry, I noticed a typo in my message. I meant "Additionally, we should not

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Sorry, I noticed a typo in my message. I meant "Additionally, we should not specify the EOL policy in this KIP" since it doesn't propose changing it. Ismael On Sun, Oct 22, 2023 at 10:56 PM Vedarth Sharma wrote: > Hi Ismael, > Thanks for the valuable feedback. > >1. No docker image

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Ismael, Thanks for the valuable feedback. 1. No docker image specific release process: This was one of our considered approaches, but we thought that docker image shouldn't block AK release. Though I agree, treating docker image as another artifact for every AK release makes much

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Vedarth, I think we shouldn't introduce any new release process that is docker specific. We should consider the software in the docker image in the same way as consider third party dependencies today - if there is a high severity CVE affecting any of them, we aim to release a new version of

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Mickael, Thanks for going through the KIP and providing valuable feedback. 1. We will support the latest LTS version of Java supported by Apache Kafka. 2. We will provide support for the last three releases. We've added a detailed example of this in the KIP under our EOL policy.

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, Overall I'm supportive of having an official docker image. I have a few questions: - Can you clarify the process of selecting the Java version? Is the proposal to only pick LTS versions? or to pick the highest version supported by Kafka? - Once a new Kafka version is released, what

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Manikumar, Thanks for the feedback! 1. We propose the addition of a new directory named "docker" at the root of the repository, where all Docker-related code will be stored. A detailed directory structure has been added in the KIP. 2. We request the creation of an Apache Kafka repository

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, Vedarth, Thanks for the KIP. 1. Can we add directory structure of Docker Image related files in Kafka repo. 2. > Steps for the Docker image release will be included in the Release Process doc of Apache Kafka Can we list down the requirements (repos, accounts) for releasing images

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Viktor, I've noticed there are two types of custom jar configurations: 1. *Type 1*: In this case, only the class name is required(e.g *authorizer.class.name **)* This can be configured by the following steps: - Mount the jar in the container.

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hey folks! We've incorporated all the points discussed into the KIP. Additionally, we've included a section detailing the Release Process of the Docker image. We value your feedback, so please take some time to review it. Once we have your inputs, we can proceed with the voting process. Thanks

[DISCUSS] KIP-975 Docker Image for Apache Kafka

Hey folks! Please feel free to ask any additional questions concerning this KIP. If there are no additional queries, shall we consider moving forward with the voting process? Thanks and regards, Vedarth

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hey Viktor! Thanks for bringing up this use case. I think we can take advantage of Docker volume for this. We can allow users to mount a folder containing the secret files. This folder can then be used to pass secrets to the container. Thanks and regards, Vedarth On Wed, Sep 27, 2023 at 4:39 PM

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, Thanks for the answer. I've seen multiple such images where environment variables are used and I think they are generally good but it's unsafe for passing around secrets, jaas configs and so on. Perhaps for secrets we could recommend using the file config provider. Then users can

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Ismael, Apologies for missing the mailing list in the last reply. Thank you for the suggestions. Just to clarify, the sizes mentioned in the previous email are of the uncompressed base images, not the resulting Apache Kafka docker images: 1. eclipse-temurin:17-jre -- 263MB (They should

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, Looks like you sent the response to me and not the mailing list, please include the mailing list in the replies. Comments below. On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal < krishna0608agar...@gmail.com> wrote: > Hi Ismael, > Thanks for the questions. > >1. We intend to

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Viktor, Thanks for the questions. 1. While the docker image outlined in KIP-975 is designed for production environments, it is equally suitable for development and testing purposes. We will furnish the docker image, allowing users the flexibility to employ it according to their

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, Thanks for the KIP. A few quick questions: 1. Since this will only be available for Kafka 3.7 in the best case, I suggest we go with Java 21 instead of Java 17. Also, we should be clear about Java version expectations. Are we allowed to change the Java version whenever we want? Or

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Viktor, Thanks for the clarification. This is a separate KIP because the scope is simpler than the graalvm docker images - there is much more to figure out when it comes to using Kafka with graalvm. Ismael On Tue, Sep 19, 2023 at 3:12 AM Viktor Somogyi-Vass wrote: > Hi Ismael, > > I'm not

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Ismael, I'm not trying to advocate against the docker image, I just pointed out that the current scoping of the KIP may be a bit too generic and thought that KIP-974 and KIP-975 were aiming for mostly the same thing and can be discussed under one umbrella. Apologies if this was rooted in a

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hello, I think this would be a wonderful improvement to the ecosystem. While Viktor is correct that most Docker pipelines eventually lead to a kubernetes deployment, that should not stop us from creating an Official Docker Image. Creating a Docker image would allow us to ensure a level of quality

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Viktor, I disagree. Docker is a very popular deployment tool and it's not only used with Kubernetes. Ismael On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass wrote: > Hi Krishna, > > I think you should merge this KIP and KIP-974 as there are overlaps as > Federico pointed out on KIP-974. I

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi Krishna, I think you should merge this KIP and KIP-974 as there are overlaps as Federico pointed out on KIP-974. I think you should keep that one as it has well defined goals (improve tests) while I feel this one is too generic. Docker is usually just a tool for either testing or Kubernetes,

[DISCUSS] KIP-975 Docker Image for Apache Kafka

Hi, Apache Kafka does not have an official docker image currently. I want to submit a KIP to publish a docker image for Apache Kafka. KIP-975: Docker Image for Apache Kafka Regards, Krishna