Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Stanislav Kozlovski]

Hey all,

As the release manager for 3.7.0, I am pretty interested to know if we
should consider this a blocker.

Do we have clarity as to whether users could practically rely on this Go
script? From a shallow look, it's only used in one line in the Dockerfile.
I guess the downside is that images extending ours would have to ship with
Golang. But in theory, once we remove it - it shouldn't be problematic
unless they extended our image, rest on the assumption that Golang was
present and used some other things in their own Dockerfile that relied on
it?

It sounds a bit minor. In the interest of the release, I would prefer we
ship with this Go script in 3.7, and change it behind the scenes in the
next release.

Thoughts?


On Wed, Dec 20, 2023 at 11:30 PM Ismael Juma  wrote:

> We should be very clear on what users can rely on when it comes to the
> docker images (i.e. what are public interfaces) and what are implementation
> details (and can be changed whenever we want). That's the only way to have
> a maintainable system. Same way we make changes to internal classes even
> though users can (and some do) rely on them.
>
> Ismael
>
> On Wed, Dec 20, 2023 at 10:55 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > Yes changes have to be merged by a committer but for this kind of
> > decisions it's best if it's seen by more than one.
> >
> > > Hmm, is this a blocker? I don't see why. It would be nice to include it
> > in 3.7 and we have time, so I'm fine with that.
> > Sure, it's not a blocker in the usual sense. But if we ship this Go
> > binary it's possible users extending our images will start depending
> > on it. Since we want to get rid of it, I'd prefer if we never shipped
> > it.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Wed, Dec 20, 2023 at 4:28 PM Ismael Juma  wrote:
> > >
> > > Hi Mickael,
> > >
> > > A couple of comments inline.
> > >
> > > On Wed, Dec 20, 2023 at 3:34 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > When you say, "we have opted to take a different approach", who is
> > > > "we"? I think this decision should be made by the committers.
> > > >
> > >
> > > Changes can only be merged by committers, so I think it's implicit that
> > at
> > > least one committer would have to agree. :) I think Vedarth was simply
> > > saying that the group working on the KIP had a new proposal that
> > addressed
> > > all the goals in a better way than the original proposal.
> > >
> > > I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
> > > > as a blocker for 3.7 as I think we need to make this decision before
> > > > releasing the docker images.
> > > >
> > >
> > > Hmm, is this a blocker? I don't see why. It would be nice to include it
> > in
> > > 3.7 and we have time, so I'm fine with that.
> > >
> > > Ismael
> >
>


-- 
Best,
Stanislav

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

We should be very clear on what users can rely on when it comes to the
docker images (i.e. what are public interfaces) and what are implementation
details (and can be changed whenever we want). That's the only way to have
a maintainable system. Same way we make changes to internal classes even
though users can (and some do) rely on them.

Ismael

On Wed, Dec 20, 2023 at 10:55 AM Mickael Maison 
wrote:

> Hi,
>
> Yes changes have to be merged by a committer but for this kind of
> decisions it's best if it's seen by more than one.
>
> > Hmm, is this a blocker? I don't see why. It would be nice to include it
> in 3.7 and we have time, so I'm fine with that.
> Sure, it's not a blocker in the usual sense. But if we ship this Go
> binary it's possible users extending our images will start depending
> on it. Since we want to get rid of it, I'd prefer if we never shipped
> it.
>
> Thanks,
> Mickael
>
>
> On Wed, Dec 20, 2023 at 4:28 PM Ismael Juma  wrote:
> >
> > Hi Mickael,
> >
> > A couple of comments inline.
> >
> > On Wed, Dec 20, 2023 at 3:34 AM Mickael Maison  >
> > wrote:
> >
> > > When you say, "we have opted to take a different approach", who is
> > > "we"? I think this decision should be made by the committers.
> > >
> >
> > Changes can only be merged by committers, so I think it's implicit that
> at
> > least one committer would have to agree. :) I think Vedarth was simply
> > saying that the group working on the KIP had a new proposal that
> addressed
> > all the goals in a better way than the original proposal.
> >
> > I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
> > > as a blocker for 3.7 as I think we need to make this decision before
> > > releasing the docker images.
> > >
> >
> > Hmm, is this a blocker? I don't see why. It would be nice to include it
> in
> > 3.7 and we have time, so I'm fine with that.
> >
> > Ismael
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Mickael Maison]

Hi,

Yes changes have to be merged by a committer but for this kind of
decisions it's best if it's seen by more than one.

> Hmm, is this a blocker? I don't see why. It would be nice to include it in 
> 3.7 and we have time, so I'm fine with that.
Sure, it's not a blocker in the usual sense. But if we ship this Go
binary it's possible users extending our images will start depending
on it. Since we want to get rid of it, I'd prefer if we never shipped
it.

Thanks,
Mickael


On Wed, Dec 20, 2023 at 4:28 PM Ismael Juma  wrote:
>
> Hi Mickael,
>
> A couple of comments inline.
>
> On Wed, Dec 20, 2023 at 3:34 AM Mickael Maison 
> wrote:
>
> > When you say, "we have opted to take a different approach", who is
> > "we"? I think this decision should be made by the committers.
> >
>
> Changes can only be merged by committers, so I think it's implicit that at
> least one committer would have to agree. :) I think Vedarth was simply
> saying that the group working on the KIP had a new proposal that addressed
> all the goals in a better way than the original proposal.
>
> I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
> > as a blocker for 3.7 as I think we need to make this decision before
> > releasing the docker images.
> >
>
> Hmm, is this a blocker? I don't see why. It would be nice to include it in
> 3.7 and we have time, so I'm fine with that.
>
> Ismael

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Mickael,

A couple of comments inline.

On Wed, Dec 20, 2023 at 3:34 AM Mickael Maison 
wrote:

> When you say, "we have opted to take a different approach", who is
> "we"? I think this decision should be made by the committers.
>

Changes can only be merged by committers, so I think it's implicit that at
least one committer would have to agree. :) I think Vedarth was simply
saying that the group working on the KIP had a new proposal that addressed
all the goals in a better way than the original proposal.

I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
> as a blocker for 3.7 as I think we need to make this decision before
> releasing the docker images.
>

Hmm, is this a blocker? I don't see why. It would be nice to include it in
3.7 and we have time, so I'm fine with that.

Ismael

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Mickael Maison]

Hi Vedarth,

Thanks for the quick reply and for the clarifications.

I agree with your analysis and the alternative approach you found
makes sense. I'll try to take a look at your PR later on today or
tomorrow.

Thanks,
Mickael

On Wed, Dec 20, 2023 at 1:33 PM Vedarth Sharma  wrote:
>
> Hi Mickael,
>
> In our previous approach startup time would have been affected if the
> utility script code was written in some other language. If we had just
> replaced the utility script with java code that would have impacted startup
> time significantly. In my local machine, it took around 50ms (on M2 silicon
> chip) just for a jvm process to start and close. Doing this around 10 times
> (number of times we were calling the utility script) would have caused
> performance issues, especially on slower machines.
>
> I agree that the decision about which approach to take should be made by
> committers. When you raised the issue I re-looked at the existing solution
> and found that if I move all the logic in the wrapper, I will be able to
> achieve the same performance without the need of any utility code like the
> golang code I had. And this wrapper code will only be executed once during
> the docker container startup instead of multiple times as we were doing
> with utility script written in golang.
>
> So the problem that this approach is solving is not just removing the
> golang code, but also removing the need of any utility script.
>
> This is just an approach that I feel is an improvement over current golang
> solution, but I would be happy to take community's feedback and opinion on
> this and we can go with what committers feels is the better solution.
>
> Thanks and regards,
> Vedarth
>
> On Wed, Dec 20, 2023 at 5:05 PM Mickael Maison 
> wrote:
>
> > Hi Vedarth,
> >
> > In the PR when I raised concerns about introducing Go to the
> > repository you responded "Startup time will get impacted though, as
> > java is significantly slower than golang and we are calling this
> > script multiple times.". It would be good to provide numbers if you
> > have, so the community can make a decision. Maybe we have multiple
> > experts with Go and the risk of introducing this language is reduced,
> > but because this was not discussed, we don't know.
> >
> > When you say, "we have opted to take a different approach", who is
> > "we"? I think this decision should be made by the committers.
> >
> > I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
> > as a blocker for 3.7 as I think we need to make this decision before
> > releasing the docker images.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Dec 20, 2023 at 10:16 AM Vedarth Sharma
> >  wrote:
> > >
> > > Hello everyone,
> > >
> > > In the course of implementing this KIP, we introduced a small piece of
> > code
> > > in golang to prepare property files for the Docker image. Our decision
> > was
> > > influenced by considerations such as performance, code testability, and a
> > > reduction in the final Docker image size, as outlined in
> > > https://github.com/apache/kafka/pull/14552#issuecomment-1855353838.
> > > However, upon reflection, we acknowledge that incorporating golang
> > > introduces a new language into the repository, potentially leading to
> > > increased maintenance overhead. In light of this, we have opted to take a
> > > different approach by introducing a Docker wrapper within the Kafka
> > > codebase. This wrapper will remove the need for the Golang code.
> > > To streamline this adjustment, we have submitted a PR:
> > > https://github.com/apache/kafka/pull/15048. The proposed changes involve
> > > the removal of the previously added golang code.
> > > This change has also been documented in the KIP.
> > >
> > > Thanks and regards,
> > > Vedarth
> > >
> > > On Thu, Oct 26, 2023 at 8:09 AM Ismael Juma  wrote:
> > >
> > > > Hi Vedarth,
> > > >
> > > >
> > > > > Local Kafka startup time (without JSA): 1.592 secs
> > > > > Local Kafka startup time (with JSA): 1.016 secs
> > > > > Local Kafka startup memory usage (without JSA): 440MB
> > > > > Local Kafka startup memory usage (with JSA): 380MB
> > > >
> > > >
> > > > This is a significant reduction in start-up time (33%) - nice!
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Oct 25, 2023 at 10:24 AM Vedarth Sharma <
> > vedarth.sha...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ismael!
> > > > >
> > > > > Thanks for bringing this to our attention.
> > > > >
> > > > > We did a small POC integrating CDS with Kafka server startup, and
> > > > > encountered positive outcomes(results are added in the KIP).
> > > > > Hence, we've decided to include the dynamically generated JSA file
> > from
> > > > the
> > > > > following workflow in the Docker image:
> > > > >
> > > > >1. Start Kafka
> > > > >2. Create a topic
> > > > >3. Produce messages
> > > > >4. Consume messages
> > > > >5. Stop Kafka
> > > > >
> > > > > Additionally, we've identified some limitations of CDS, which have
> 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Mickael,

In our previous approach startup time would have been affected if the
utility script code was written in some other language. If we had just
replaced the utility script with java code that would have impacted startup
time significantly. In my local machine, it took around 50ms (on M2 silicon
chip) just for a jvm process to start and close. Doing this around 10 times
(number of times we were calling the utility script) would have caused
performance issues, especially on slower machines.

I agree that the decision about which approach to take should be made by
committers. When you raised the issue I re-looked at the existing solution
and found that if I move all the logic in the wrapper, I will be able to
achieve the same performance without the need of any utility code like the
golang code I had. And this wrapper code will only be executed once during
the docker container startup instead of multiple times as we were doing
with utility script written in golang.

So the problem that this approach is solving is not just removing the
golang code, but also removing the need of any utility script.

This is just an approach that I feel is an improvement over current golang
solution, but I would be happy to take community's feedback and opinion on
this and we can go with what committers feels is the better solution.

Thanks and regards,
Vedarth

On Wed, Dec 20, 2023 at 5:05 PM Mickael Maison 
wrote:

> Hi Vedarth,
>
> In the PR when I raised concerns about introducing Go to the
> repository you responded "Startup time will get impacted though, as
> java is significantly slower than golang and we are calling this
> script multiple times.". It would be good to provide numbers if you
> have, so the community can make a decision. Maybe we have multiple
> experts with Go and the risk of introducing this language is reduced,
> but because this was not discussed, we don't know.
>
> When you say, "we have opted to take a different approach", who is
> "we"? I think this decision should be made by the committers.
>
> I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
> as a blocker for 3.7 as I think we need to make this decision before
> releasing the docker images.
>
> Thanks,
> Mickael
>
> On Wed, Dec 20, 2023 at 10:16 AM Vedarth Sharma
>  wrote:
> >
> > Hello everyone,
> >
> > In the course of implementing this KIP, we introduced a small piece of
> code
> > in golang to prepare property files for the Docker image. Our decision
> was
> > influenced by considerations such as performance, code testability, and a
> > reduction in the final Docker image size, as outlined in
> > https://github.com/apache/kafka/pull/14552#issuecomment-1855353838.
> > However, upon reflection, we acknowledge that incorporating golang
> > introduces a new language into the repository, potentially leading to
> > increased maintenance overhead. In light of this, we have opted to take a
> > different approach by introducing a Docker wrapper within the Kafka
> > codebase. This wrapper will remove the need for the Golang code.
> > To streamline this adjustment, we have submitted a PR:
> > https://github.com/apache/kafka/pull/15048. The proposed changes involve
> > the removal of the previously added golang code.
> > This change has also been documented in the KIP.
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Thu, Oct 26, 2023 at 8:09 AM Ismael Juma  wrote:
> >
> > > Hi Vedarth,
> > >
> > >
> > > > Local Kafka startup time (without JSA): 1.592 secs
> > > > Local Kafka startup time (with JSA): 1.016 secs
> > > > Local Kafka startup memory usage (without JSA): 440MB
> > > > Local Kafka startup memory usage (with JSA): 380MB
> > >
> > >
> > > This is a significant reduction in start-up time (33%) - nice!
> > >
> > > Ismael
> > >
> > > On Wed, Oct 25, 2023 at 10:24 AM Vedarth Sharma <
> vedarth.sha...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ismael!
> > > >
> > > > Thanks for bringing this to our attention.
> > > >
> > > > We did a small POC integrating CDS with Kafka server startup, and
> > > > encountered positive outcomes(results are added in the KIP).
> > > > Hence, we've decided to include the dynamically generated JSA file
> from
> > > the
> > > > following workflow in the Docker image:
> > > >
> > > >1. Start Kafka
> > > >2. Create a topic
> > > >3. Produce messages
> > > >4. Consume messages
> > > >5. Stop Kafka
> > > >
> > > > Additionally, we've identified some limitations of CDS, which have
> also
> > > > been detailed in the KIP.
> > > >
> > > > Thanks and regards,
> > > > Vedarth
> > > >
> > > > On Wed, Oct 25, 2023 at 10:56 AM Ismael Juma 
> wrote:
> > > >
> > > > > The reference I meant to include:
> > > > >
> > > > >
> https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html
> > > > >
> > > > > On Tue, Oct 24, 2023, 10:25 PM Ismael Juma 
> wrote:
> > > > >
> > > > > > Hi Krishna,
> > > > > >
> > > > > > One last question from me, did we confuse using AppCDS or Dynamic
> >

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Mickael Maison]

Hi Vedarth,

In the PR when I raised concerns about introducing Go to the
repository you responded "Startup time will get impacted though, as
java is significantly slower than golang and we are calling this
script multiple times.". It would be good to provide numbers if you
have, so the community can make a decision. Maybe we have multiple
experts with Go and the risk of introducing this language is reduced,
but because this was not discussed, we don't know.

When you say, "we have opted to take a different approach", who is
"we"? I think this decision should be made by the committers.

I marked the Jira (https://issues.apache.org/jira/browse/KAFKA-16016)
as a blocker for 3.7 as I think we need to make this decision before
releasing the docker images.

Thanks,
Mickael

On Wed, Dec 20, 2023 at 10:16 AM Vedarth Sharma
 wrote:
>
> Hello everyone,
>
> In the course of implementing this KIP, we introduced a small piece of code
> in golang to prepare property files for the Docker image. Our decision was
> influenced by considerations such as performance, code testability, and a
> reduction in the final Docker image size, as outlined in
> https://github.com/apache/kafka/pull/14552#issuecomment-1855353838.
> However, upon reflection, we acknowledge that incorporating golang
> introduces a new language into the repository, potentially leading to
> increased maintenance overhead. In light of this, we have opted to take a
> different approach by introducing a Docker wrapper within the Kafka
> codebase. This wrapper will remove the need for the Golang code.
> To streamline this adjustment, we have submitted a PR:
> https://github.com/apache/kafka/pull/15048. The proposed changes involve
> the removal of the previously added golang code.
> This change has also been documented in the KIP.
>
> Thanks and regards,
> Vedarth
>
> On Thu, Oct 26, 2023 at 8:09 AM Ismael Juma  wrote:
>
> > Hi Vedarth,
> >
> >
> > > Local Kafka startup time (without JSA): 1.592 secs
> > > Local Kafka startup time (with JSA): 1.016 secs
> > > Local Kafka startup memory usage (without JSA): 440MB
> > > Local Kafka startup memory usage (with JSA): 380MB
> >
> >
> > This is a significant reduction in start-up time (33%) - nice!
> >
> > Ismael
> >
> > On Wed, Oct 25, 2023 at 10:24 AM Vedarth Sharma 
> > wrote:
> >
> > > Hi Ismael!
> > >
> > > Thanks for bringing this to our attention.
> > >
> > > We did a small POC integrating CDS with Kafka server startup, and
> > > encountered positive outcomes(results are added in the KIP).
> > > Hence, we've decided to include the dynamically generated JSA file from
> > the
> > > following workflow in the Docker image:
> > >
> > >1. Start Kafka
> > >2. Create a topic
> > >3. Produce messages
> > >4. Consume messages
> > >5. Stop Kafka
> > >
> > > Additionally, we've identified some limitations of CDS, which have also
> > > been detailed in the KIP.
> > >
> > > Thanks and regards,
> > > Vedarth
> > >
> > > On Wed, Oct 25, 2023 at 10:56 AM Ismael Juma  wrote:
> > >
> > > > The reference I meant to include:
> > > >
> > > > https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html
> > > >
> > > > On Tue, Oct 24, 2023, 10:25 PM Ismael Juma  wrote:
> > > >
> > > > > Hi Krishna,
> > > > >
> > > > > One last question from me, did we confuse using AppCDS or Dynamic
> > CDS?
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > > > On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal <
> > > > > krishna0608agar...@gmail.com> wrote:
> > > > >
> > > > >> Hi,
> > > > >> Thanks for the insightful feedback on this KIP. If there are no
> > > further
> > > > >> questions, I'm considering wrapping up this discussion thread. We'll
> > > be
> > > > >> moving into the voting process in the next couple of days. Your
> > > > continued
> > > > >> input is greatly appreciated!
> > > > >>
> > > > >> Regards,
> > > > >> Krishna
> > > > >>
> > > > >> On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal <
> > > > >> krishna0608agar...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> > Apache Kafka does not have an official docker image currently.
> > > > >> > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > > >> >
> > > > >> > KIP-975: Docker Image for Apache Kafka
> > > > >> > <
> > > > >>
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > >> >
> > > > >> >
> > > > >> > Regards,
> > > > >> > Krishna
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hello everyone,

In the course of implementing this KIP, we introduced a small piece of code
in golang to prepare property files for the Docker image. Our decision was
influenced by considerations such as performance, code testability, and a
reduction in the final Docker image size, as outlined in
https://github.com/apache/kafka/pull/14552#issuecomment-1855353838.
However, upon reflection, we acknowledge that incorporating golang
introduces a new language into the repository, potentially leading to
increased maintenance overhead. In light of this, we have opted to take a
different approach by introducing a Docker wrapper within the Kafka
codebase. This wrapper will remove the need for the Golang code.
To streamline this adjustment, we have submitted a PR:
https://github.com/apache/kafka/pull/15048. The proposed changes involve
the removal of the previously added golang code.
This change has also been documented in the KIP.

Thanks and regards,
Vedarth

On Thu, Oct 26, 2023 at 8:09 AM Ismael Juma  wrote:

> Hi Vedarth,
>
>
> > Local Kafka startup time (without JSA): 1.592 secs
> > Local Kafka startup time (with JSA): 1.016 secs
> > Local Kafka startup memory usage (without JSA): 440MB
> > Local Kafka startup memory usage (with JSA): 380MB
>
>
> This is a significant reduction in start-up time (33%) - nice!
>
> Ismael
>
> On Wed, Oct 25, 2023 at 10:24 AM Vedarth Sharma 
> wrote:
>
> > Hi Ismael!
> >
> > Thanks for bringing this to our attention.
> >
> > We did a small POC integrating CDS with Kafka server startup, and
> > encountered positive outcomes(results are added in the KIP).
> > Hence, we've decided to include the dynamically generated JSA file from
> the
> > following workflow in the Docker image:
> >
> >1. Start Kafka
> >2. Create a topic
> >3. Produce messages
> >4. Consume messages
> >5. Stop Kafka
> >
> > Additionally, we've identified some limitations of CDS, which have also
> > been detailed in the KIP.
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Wed, Oct 25, 2023 at 10:56 AM Ismael Juma  wrote:
> >
> > > The reference I meant to include:
> > >
> > > https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html
> > >
> > > On Tue, Oct 24, 2023, 10:25 PM Ismael Juma  wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > One last question from me, did we confuse using AppCDS or Dynamic
> CDS?
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > > > On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com> wrote:
> > > >
> > > >> Hi,
> > > >> Thanks for the insightful feedback on this KIP. If there are no
> > further
> > > >> questions, I'm considering wrapping up this discussion thread. We'll
> > be
> > > >> moving into the voting process in the next couple of days. Your
> > > continued
> > > >> input is greatly appreciated!
> > > >>
> > > >> Regards,
> > > >> Krishna
> > > >>
> > > >> On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal <
> > > >> krishna0608agar...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> > Apache Kafka does not have an official docker image currently.
> > > >> > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > >> >
> > > >> > KIP-975: Docker Image for Apache Kafka
> > > >> > <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > >> >
> > > >> >
> > > >> > Regards,
> > > >> > Krishna
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Vedarth,


> Local Kafka startup time (without JSA): 1.592 secs
> Local Kafka startup time (with JSA): 1.016 secs
> Local Kafka startup memory usage (without JSA): 440MB
> Local Kafka startup memory usage (with JSA): 380MB


This is a significant reduction in start-up time (33%) - nice!

Ismael

On Wed, Oct 25, 2023 at 10:24 AM Vedarth Sharma 
wrote:

> Hi Ismael!
>
> Thanks for bringing this to our attention.
>
> We did a small POC integrating CDS with Kafka server startup, and
> encountered positive outcomes(results are added in the KIP).
> Hence, we've decided to include the dynamically generated JSA file from the
> following workflow in the Docker image:
>
>1. Start Kafka
>2. Create a topic
>3. Produce messages
>4. Consume messages
>5. Stop Kafka
>
> Additionally, we've identified some limitations of CDS, which have also
> been detailed in the KIP.
>
> Thanks and regards,
> Vedarth
>
> On Wed, Oct 25, 2023 at 10:56 AM Ismael Juma  wrote:
>
> > The reference I meant to include:
> >
> > https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html
> >
> > On Tue, Oct 24, 2023, 10:25 PM Ismael Juma  wrote:
> >
> > > Hi Krishna,
> > >
> > > One last question from me, did we confuse using AppCDS or Dynamic CDS?
> > >
> > > Thanks,
> > > Ismael
> > >
> > > On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > >> Hi,
> > >> Thanks for the insightful feedback on this KIP. If there are no
> further
> > >> questions, I'm considering wrapping up this discussion thread. We'll
> be
> > >> moving into the voting process in the next couple of days. Your
> > continued
> > >> input is greatly appreciated!
> > >>
> > >> Regards,
> > >> Krishna
> > >>
> > >> On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal <
> > >> krishna0608agar...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> > Apache Kafka does not have an official docker image currently.
> > >> > I want to submit a KIP to publish a docker image for Apache Kafka.
> > >> >
> > >> > KIP-975: Docker Image for Apache Kafka
> > >> > <
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > >> >
> > >> >
> > >> > Regards,
> > >> > Krishna
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Ismael!

Thanks for bringing this to our attention.

We did a small POC integrating CDS with Kafka server startup, and
encountered positive outcomes(results are added in the KIP).
Hence, we've decided to include the dynamically generated JSA file from the
following workflow in the Docker image:

   1. Start Kafka
   2. Create a topic
   3. Produce messages
   4. Consume messages
   5. Stop Kafka

Additionally, we've identified some limitations of CDS, which have also
been detailed in the KIP.

Thanks and regards,
Vedarth

On Wed, Oct 25, 2023 at 10:56 AM Ismael Juma  wrote:

> The reference I meant to include:
>
> https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html
>
> On Tue, Oct 24, 2023, 10:25 PM Ismael Juma  wrote:
>
> > Hi Krishna,
> >
> > One last question from me, did we confuse using AppCDS or Dynamic CDS?
> >
> > Thanks,
> > Ismael
> >
> > On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> >> Hi,
> >> Thanks for the insightful feedback on this KIP. If there are no further
> >> questions, I'm considering wrapping up this discussion thread. We'll be
> >> moving into the voting process in the next couple of days. Your
> continued
> >> input is greatly appreciated!
> >>
> >> Regards,
> >> Krishna
> >>
> >> On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal <
> >> krishna0608agar...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > Apache Kafka does not have an official docker image currently.
> >> > I want to submit a KIP to publish a docker image for Apache Kafka.
> >> >
> >> > KIP-975: Docker Image for Apache Kafka
> >> > <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >> >
> >> >
> >> > Regards,
> >> > Krishna
> >> >
> >>
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

The reference I meant to include:

https://docs.oracle.com/en/java/javase/17/vm/class-data-sharing.html

On Tue, Oct 24, 2023, 10:25 PM Ismael Juma  wrote:

> Hi Krishna,
>
> One last question from me, did we confuse using AppCDS or Dynamic CDS?
>
> Thanks,
> Ismael
>
> On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
>> Hi,
>> Thanks for the insightful feedback on this KIP. If there are no further
>> questions, I'm considering wrapping up this discussion thread. We'll be
>> moving into the voting process in the next couple of days. Your continued
>> input is greatly appreciated!
>>
>> Regards,
>> Krishna
>>
>> On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal <
>> krishna0608agar...@gmail.com>
>> wrote:
>>
>> > Hi,
>> > Apache Kafka does not have an official docker image currently.
>> > I want to submit a KIP to publish a docker image for Apache Kafka.
>> >
>> > KIP-975: Docker Image for Apache Kafka
>> > <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>> >
>> >
>> > Regards,
>> > Krishna
>> >
>>
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Krishna,

One last question from me, did we confuse using AppCDS or Dynamic CDS?

Thanks,
Ismael

On Tue, Oct 24, 2023, 9:54 PM Krishna Agarwal 
wrote:

> Hi,
> Thanks for the insightful feedback on this KIP. If there are no further
> questions, I'm considering wrapping up this discussion thread. We'll be
> moving into the voting process in the next couple of days. Your continued
> input is greatly appreciated!
>
> Regards,
> Krishna
>
> On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal <
> krishna0608agar...@gmail.com>
> wrote:
>
> > Hi,
> > Apache Kafka does not have an official docker image currently.
> > I want to submit a KIP to publish a docker image for Apache Kafka.
> >
> > KIP-975: Docker Image for Apache Kafka
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >
> >
> > Regards,
> > Krishna
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Krishna Agarwal]

Hi,
Thanks for the insightful feedback on this KIP. If there are no further
questions, I'm considering wrapping up this discussion thread. We'll be
moving into the voting process in the next couple of days. Your continued
input is greatly appreciated!

Regards,
Krishna

On Fri, Sep 8, 2023 at 1:27 PM Krishna Agarwal 
wrote:

> Hi,
> Apache Kafka does not have an official docker image currently.
> I want to submit a KIP to publish a docker image for Apache Kafka.
>
> KIP-975: Docker Image for Apache Kafka
> 
>
> Regards,
> Krishna
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Ismael,

Thanks for the correction.

We will be following the same EOL policy as Apache Kafka, hence we have
removed it from the KIP.

Thanks and regards,
Vedarth

On Mon, Oct 23, 2023 at 12:17 PM Ismael Juma  wrote:

> Sorry, I noticed a typo in my message. I meant "Additionally, we should not
> specify the EOL policy in this KIP" since it doesn't propose changing it.
>
> Ismael
>
> On Sun, Oct 22, 2023 at 10:56 PM Vedarth Sharma 
> wrote:
>
> > Hi Ismael,
> > Thanks for the valuable feedback.
> >
> >1. No docker image specific release process: This was one of our
> >considered approaches, but we thought that docker image shouldn't
> block
> > AK
> >release. Though I agree, treating docker image as another artifact for
> >every AK release makes much more sense. Hence, releasing a new version
> > of
> >Kafka for the affected branch in such a scenario is a much cleaner
> >approach. Added this as the accepted approach in the KIP.
> >2. EOL policy: Updated in the KIP.
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Sun, Oct 22, 2023 at 11:20 PM Ismael Juma  wrote:
> >
> > > Hi Vedarth,
> > >
> > > I think we shouldn't introduce any new release process that is docker
> > > specific. We should consider the software in the docker image in the
> same
> > > way as consider third party dependencies today - if there is a high
> > > severity CVE affecting any of them, we aim to release a new version of
> > > Kafka for the affected branch. It would include the latest Kafka code
> > from
> > > the branch.
> > >
> > > Additionally, we should specify the EOL policy in this KIP - we are not
> > > changing it as part of it. One interesting detail is that the release
> > > document claims we support the last 3 releases, but the reality has
> been
> > a
> > > bit different - we tend to support the 2 most recent releases unless
> > it's a
> > > high severity CVE in Kafka itself (these tend to be much rarer,
> > > thankfully).
> > >
> > > Ismael
> > >
> > > On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma <
> vedarth.sha...@gmail.com>
> > > wrote:
> > >
> > > > Hi Mickael,
> > > > Thanks for going through the KIP and providing valuable feedback.
> > > >
> > > >1. We will support the latest LTS version of Java supported by
> > Apache
> > > >Kafka.
> > > >2. We will provide support for the last three releases. We've
> added
> > a
> > > >detailed example of this in the KIP under our EOL policy.
> > > >3. We can establish a nightly cron job using GitHub Actions and
> > > leverage
> > > >an open-source vulnerability scanning tool like trivy (
> > > >https://github.com/aquasecurity/trivy), to get vulnerability
> > reports
> > > on
> > > >all supported images. This tool offers a straightforward way to
> > > > integrate
> > > >vulnerability checks directly into our GitHub Actions workflow.
> > > >4. That's a good suggestion to have a GitHub Actions workflow. We
> > will
> > > >implement a GitHub Actions workflow to automate the build and
> > testing
> > > >process.
> > > >5. Regarding the release process, we observed that there isn't an
> > > >existing CI/CD pipeline. We can consider the addition of a GitHub
> > > > workflow
> > > >to facilitate the release process.
> > > >
> > > > Please let us know your thoughts on the above.
> > > >
> > > > Thanks and regards,
> > > > Vedarth
> > > >
> > > > On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Krishna,
> > > > >
> > > > > Overall I'm supportive of having an official docker image.
> > > > > I have a few questions:
> > > > > - Can you clarify the process of selecting the Java version? Is the
> > > > > proposal to only pick LTS versions? or to pick the highest version
> > > > > supported by Kafka?
> > > > > - Once a new Kafka version is released, what happens to the image
> > > > > containing the previous release? Do we expect to still update it in
> > > > > case of CVEs? If so for how long?
> > > > > - How will we get notified that the base image has a CVE?
> > > > > - Rather than having scripts PMC members have to run from their
> > > > > machines, would it e possible to have a Jenkins job or GitHub
> action?
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
> > > > >  wrote:
> > > > > >
> > > > > > Hi Manikumar,
> > > > > >
> > > > > > Thanks for the feedback!
> > > > > >
> > > > > > 1. We propose the addition of a new directory named "docker" at
> the
> > > > root
> > > > > of
> > > > > > the repository, where all Docker-related code will be stored. A
> > > > detailed
> > > > > > directory structure has been added in the KIP.
> > > > > > 2. We request the creation of an Apache Kafka repository
> > > (apache/kafka)
> > > > > on
> > > > > > DockerHub, to be administered under the The Apache Software
> > > Foundation
> > > > > > 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Sorry, I noticed a typo in my message. I meant "Additionally, we should not
specify the EOL policy in this KIP" since it doesn't propose changing it.

Ismael

On Sun, Oct 22, 2023 at 10:56 PM Vedarth Sharma 
wrote:

> Hi Ismael,
> Thanks for the valuable feedback.
>
>1. No docker image specific release process: This was one of our
>considered approaches, but we thought that docker image shouldn't block
> AK
>release. Though I agree, treating docker image as another artifact for
>every AK release makes much more sense. Hence, releasing a new version
> of
>Kafka for the affected branch in such a scenario is a much cleaner
>approach. Added this as the accepted approach in the KIP.
>2. EOL policy: Updated in the KIP.
>
> Thanks and regards,
> Vedarth
>
> On Sun, Oct 22, 2023 at 11:20 PM Ismael Juma  wrote:
>
> > Hi Vedarth,
> >
> > I think we shouldn't introduce any new release process that is docker
> > specific. We should consider the software in the docker image in the same
> > way as consider third party dependencies today - if there is a high
> > severity CVE affecting any of them, we aim to release a new version of
> > Kafka for the affected branch. It would include the latest Kafka code
> from
> > the branch.
> >
> > Additionally, we should specify the EOL policy in this KIP - we are not
> > changing it as part of it. One interesting detail is that the release
> > document claims we support the last 3 releases, but the reality has been
> a
> > bit different - we tend to support the 2 most recent releases unless
> it's a
> > high severity CVE in Kafka itself (these tend to be much rarer,
> > thankfully).
> >
> > Ismael
> >
> > On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma 
> > wrote:
> >
> > > Hi Mickael,
> > > Thanks for going through the KIP and providing valuable feedback.
> > >
> > >1. We will support the latest LTS version of Java supported by
> Apache
> > >Kafka.
> > >2. We will provide support for the last three releases. We've added
> a
> > >detailed example of this in the KIP under our EOL policy.
> > >3. We can establish a nightly cron job using GitHub Actions and
> > leverage
> > >an open-source vulnerability scanning tool like trivy (
> > >https://github.com/aquasecurity/trivy), to get vulnerability
> reports
> > on
> > >all supported images. This tool offers a straightforward way to
> > > integrate
> > >vulnerability checks directly into our GitHub Actions workflow.
> > >4. That's a good suggestion to have a GitHub Actions workflow. We
> will
> > >implement a GitHub Actions workflow to automate the build and
> testing
> > >process.
> > >5. Regarding the release process, we observed that there isn't an
> > >existing CI/CD pipeline. We can consider the addition of a GitHub
> > > workflow
> > >to facilitate the release process.
> > >
> > > Please let us know your thoughts on the above.
> > >
> > > Thanks and regards,
> > > Vedarth
> > >
> > > On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > Overall I'm supportive of having an official docker image.
> > > > I have a few questions:
> > > > - Can you clarify the process of selecting the Java version? Is the
> > > > proposal to only pick LTS versions? or to pick the highest version
> > > > supported by Kafka?
> > > > - Once a new Kafka version is released, what happens to the image
> > > > containing the previous release? Do we expect to still update it in
> > > > case of CVEs? If so for how long?
> > > > - How will we get notified that the base image has a CVE?
> > > > - Rather than having scripts PMC members have to run from their
> > > > machines, would it e possible to have a Jenkins job or GitHub action?
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
> > > >  wrote:
> > > > >
> > > > > Hi Manikumar,
> > > > >
> > > > > Thanks for the feedback!
> > > > >
> > > > > 1. We propose the addition of a new directory named "docker" at the
> > > root
> > > > of
> > > > > the repository, where all Docker-related code will be stored. A
> > > detailed
> > > > > directory structure has been added in the KIP.
> > > > > 2. We request the creation of an Apache Kafka repository
> > (apache/kafka)
> > > > on
> > > > > DockerHub, to be administered under the The Apache Software
> > Foundation
> > > > > . The PMC members should have the
> > > > > necessary permissions for pushing updates to the docker repo.
> > > > >
> > > > > Thanks and regards,
> > > > > Vedarth
> > > > >
> > > > >
> > > > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar <
> manikumar.re...@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > > Hi Krishna, Vedarth,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > 1. Can we add directory structure of Docker Image related files
> in
> > > > Kafka
> > > > > > 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Ismael,
Thanks for the valuable feedback.

   1. No docker image specific release process: This was one of our
   considered approaches, but we thought that docker image shouldn't block AK
   release. Though I agree, treating docker image as another artifact for
   every AK release makes much more sense. Hence, releasing a new version of
   Kafka for the affected branch in such a scenario is a much cleaner
   approach. Added this as the accepted approach in the KIP.
   2. EOL policy: Updated in the KIP.

Thanks and regards,
Vedarth

On Sun, Oct 22, 2023 at 11:20 PM Ismael Juma  wrote:

> Hi Vedarth,
>
> I think we shouldn't introduce any new release process that is docker
> specific. We should consider the software in the docker image in the same
> way as consider third party dependencies today - if there is a high
> severity CVE affecting any of them, we aim to release a new version of
> Kafka for the affected branch. It would include the latest Kafka code from
> the branch.
>
> Additionally, we should specify the EOL policy in this KIP - we are not
> changing it as part of it. One interesting detail is that the release
> document claims we support the last 3 releases, but the reality has been a
> bit different - we tend to support the 2 most recent releases unless it's a
> high severity CVE in Kafka itself (these tend to be much rarer,
> thankfully).
>
> Ismael
>
> On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma 
> wrote:
>
> > Hi Mickael,
> > Thanks for going through the KIP and providing valuable feedback.
> >
> >1. We will support the latest LTS version of Java supported by Apache
> >Kafka.
> >2. We will provide support for the last three releases. We've added a
> >detailed example of this in the KIP under our EOL policy.
> >3. We can establish a nightly cron job using GitHub Actions and
> leverage
> >an open-source vulnerability scanning tool like trivy (
> >https://github.com/aquasecurity/trivy), to get vulnerability reports
> on
> >all supported images. This tool offers a straightforward way to
> > integrate
> >vulnerability checks directly into our GitHub Actions workflow.
> >4. That's a good suggestion to have a GitHub Actions workflow. We will
> >implement a GitHub Actions workflow to automate the build and testing
> >process.
> >5. Regarding the release process, we observed that there isn't an
> >existing CI/CD pipeline. We can consider the addition of a GitHub
> > workflow
> >to facilitate the release process.
> >
> > Please let us know your thoughts on the above.
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison  >
> > wrote:
> >
> > > Hi Krishna,
> > >
> > > Overall I'm supportive of having an official docker image.
> > > I have a few questions:
> > > - Can you clarify the process of selecting the Java version? Is the
> > > proposal to only pick LTS versions? or to pick the highest version
> > > supported by Kafka?
> > > - Once a new Kafka version is released, what happens to the image
> > > containing the previous release? Do we expect to still update it in
> > > case of CVEs? If so for how long?
> > > - How will we get notified that the base image has a CVE?
> > > - Rather than having scripts PMC members have to run from their
> > > machines, would it e possible to have a Jenkins job or GitHub action?
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
> > >  wrote:
> > > >
> > > > Hi Manikumar,
> > > >
> > > > Thanks for the feedback!
> > > >
> > > > 1. We propose the addition of a new directory named "docker" at the
> > root
> > > of
> > > > the repository, where all Docker-related code will be stored. A
> > detailed
> > > > directory structure has been added in the KIP.
> > > > 2. We request the creation of an Apache Kafka repository
> (apache/kafka)
> > > on
> > > > DockerHub, to be administered under the The Apache Software
> Foundation
> > > > . The PMC members should have the
> > > > necessary permissions for pushing updates to the docker repo.
> > > >
> > > > Thanks and regards,
> > > > Vedarth
> > > >
> > > >
> > > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar  >
> > > wrote:
> > > >
> > > > > Hi Krishna, Vedarth,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > 1. Can we add directory structure of Docker Image related files in
> > > Kafka
> > > > > repo.
> > > > >
> > > > > 2. > Steps for the Docker image release will be included in the
> > Release
> > > > > Process doc of Apache Kafka
> > > > >
> > > > > Can we list down the requirements (repos, accounts) for releasing
> > > images to
> > > > > docker hub. I am mainly asking because PMC needs to request docker
> > hub
> > > > > access/repos.
> > > > > I can help in getting required repos/accounts.
> > > > > https://infra.apache.org/docker-hub-policy.html
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > > >

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Vedarth,

I think we shouldn't introduce any new release process that is docker
specific. We should consider the software in the docker image in the same
way as consider third party dependencies today - if there is a high
severity CVE affecting any of them, we aim to release a new version of
Kafka for the affected branch. It would include the latest Kafka code from
the branch.

Additionally, we should specify the EOL policy in this KIP - we are not
changing it as part of it. One interesting detail is that the release
document claims we support the last 3 releases, but the reality has been a
bit different - we tend to support the 2 most recent releases unless it's a
high severity CVE in Kafka itself (these tend to be much rarer, thankfully).

Ismael

On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma 
wrote:

> Hi Mickael,
> Thanks for going through the KIP and providing valuable feedback.
>
>1. We will support the latest LTS version of Java supported by Apache
>Kafka.
>2. We will provide support for the last three releases. We've added a
>detailed example of this in the KIP under our EOL policy.
>3. We can establish a nightly cron job using GitHub Actions and leverage
>an open-source vulnerability scanning tool like trivy (
>https://github.com/aquasecurity/trivy), to get vulnerability reports on
>all supported images. This tool offers a straightforward way to
> integrate
>vulnerability checks directly into our GitHub Actions workflow.
>4. That's a good suggestion to have a GitHub Actions workflow. We will
>implement a GitHub Actions workflow to automate the build and testing
>process.
>5. Regarding the release process, we observed that there isn't an
>existing CI/CD pipeline. We can consider the addition of a GitHub
> workflow
>to facilitate the release process.
>
> Please let us know your thoughts on the above.
>
> Thanks and regards,
> Vedarth
>
> On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison 
> wrote:
>
> > Hi Krishna,
> >
> > Overall I'm supportive of having an official docker image.
> > I have a few questions:
> > - Can you clarify the process of selecting the Java version? Is the
> > proposal to only pick LTS versions? or to pick the highest version
> > supported by Kafka?
> > - Once a new Kafka version is released, what happens to the image
> > containing the previous release? Do we expect to still update it in
> > case of CVEs? If so for how long?
> > - How will we get notified that the base image has a CVE?
> > - Rather than having scripts PMC members have to run from their
> > machines, would it e possible to have a Jenkins job or GitHub action?
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
> >  wrote:
> > >
> > > Hi Manikumar,
> > >
> > > Thanks for the feedback!
> > >
> > > 1. We propose the addition of a new directory named "docker" at the
> root
> > of
> > > the repository, where all Docker-related code will be stored. A
> detailed
> > > directory structure has been added in the KIP.
> > > 2. We request the creation of an Apache Kafka repository (apache/kafka)
> > on
> > > DockerHub, to be administered under the The Apache Software Foundation
> > > . The PMC members should have the
> > > necessary permissions for pushing updates to the docker repo.
> > >
> > > Thanks and regards,
> > > Vedarth
> > >
> > >
> > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar 
> > wrote:
> > >
> > > > Hi Krishna, Vedarth,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > 1. Can we add directory structure of Docker Image related files in
> > Kafka
> > > > repo.
> > > >
> > > > 2. > Steps for the Docker image release will be included in the
> Release
> > > > Process doc of Apache Kafka
> > > >
> > > > Can we list down the requirements (repos, accounts) for releasing
> > images to
> > > > docker hub. I am mainly asking because PMC needs to request docker
> hub
> > > > access/repos.
> > > > I can help in getting required repos/accounts.
> > > > https://infra.apache.org/docker-hub-policy.html
> > > >
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com> wrote:
> > > >
> > > > > Hi Viktor,
> > > > >
> > > > > I've noticed there are two types of custom jar configurations:
> > > > >
> > > > >1. *Type 1*: In this case, only the class name is required(e.g
> > > > > *authorizer.class.name
> > > > >**)* This can be configured by
> the
> > > > >following steps:
> > > > >   - Mount the jar in the container.
> > > > >   - Configure the *CLASSPATH* environment variable (used by
> > > > >   *kafka-run-class.sh*) by providing the mounted path to it.
> > This can
> > > > >   be passed as an environment variable to the docker container.
> > > > >2. *Type 2*: Here, in addition to the class name, classpath can
> > also
> > > > be
> > > > >config

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Mickael,
Thanks for going through the KIP and providing valuable feedback.

   1. We will support the latest LTS version of Java supported by Apache
   Kafka.
   2. We will provide support for the last three releases. We've added a
   detailed example of this in the KIP under our EOL policy.
   3. We can establish a nightly cron job using GitHub Actions and leverage
   an open-source vulnerability scanning tool like trivy (
   https://github.com/aquasecurity/trivy), to get vulnerability reports on
   all supported images. This tool offers a straightforward way to integrate
   vulnerability checks directly into our GitHub Actions workflow.
   4. That's a good suggestion to have a GitHub Actions workflow. We will
   implement a GitHub Actions workflow to automate the build and testing
   process.
   5. Regarding the release process, we observed that there isn't an
   existing CI/CD pipeline. We can consider the addition of a GitHub workflow
   to facilitate the release process.

Please let us know your thoughts on the above.

Thanks and regards,
Vedarth

On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison 
wrote:

> Hi Krishna,
>
> Overall I'm supportive of having an official docker image.
> I have a few questions:
> - Can you clarify the process of selecting the Java version? Is the
> proposal to only pick LTS versions? or to pick the highest version
> supported by Kafka?
> - Once a new Kafka version is released, what happens to the image
> containing the previous release? Do we expect to still update it in
> case of CVEs? If so for how long?
> - How will we get notified that the base image has a CVE?
> - Rather than having scripts PMC members have to run from their
> machines, would it e possible to have a Jenkins job or GitHub action?
>
> Thanks,
> Mickael
>
>
>
> On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
>  wrote:
> >
> > Hi Manikumar,
> >
> > Thanks for the feedback!
> >
> > 1. We propose the addition of a new directory named "docker" at the root
> of
> > the repository, where all Docker-related code will be stored. A detailed
> > directory structure has been added in the KIP.
> > 2. We request the creation of an Apache Kafka repository (apache/kafka)
> on
> > DockerHub, to be administered under the The Apache Software Foundation
> > . The PMC members should have the
> > necessary permissions for pushing updates to the docker repo.
> >
> > Thanks and regards,
> > Vedarth
> >
> >
> > On Fri, Oct 20, 2023 at 2:44 PM Manikumar 
> wrote:
> >
> > > Hi Krishna, Vedarth,
> > >
> > > Thanks for the KIP.
> > >
> > > 1. Can we add directory structure of Docker Image related files in
> Kafka
> > > repo.
> > >
> > > 2. > Steps for the Docker image release will be included in the Release
> > > Process doc of Apache Kafka
> > >
> > > Can we list down the requirements (repos, accounts) for releasing
> images to
> > > docker hub. I am mainly asking because PMC needs to request docker hub
> > > access/repos.
> > > I can help in getting required repos/accounts.
> > > https://infra.apache.org/docker-hub-policy.html
> > >
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > > > Hi Viktor,
> > > >
> > > > I've noticed there are two types of custom jar configurations:
> > > >
> > > >1. *Type 1*: In this case, only the class name is required(e.g
> > > > *authorizer.class.name
> > > >**)* This can be configured by the
> > > >following steps:
> > > >   - Mount the jar in the container.
> > > >   - Configure the *CLASSPATH* environment variable (used by
> > > >   *kafka-run-class.sh*) by providing the mounted path to it.
> This can
> > > >   be passed as an environment variable to the docker container.
> > > >2. *Type 2*: Here, in addition to the class name, classpath can
> also
> > > be
> > > >configured (eg *remote.log.metadata.manager.class.name
> > > > *and
> > > >*remote.log.metadata.manager.class.path*). This can be configured
> by
> > > the
> > > >following steps:
> > > >   - Mount the jar in the container.
> > > >   - Configure the respective *class.path* property.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com> wrote:
> > > >
> > > > > Hi Viktor,
> > > > > Thanks for the questions.
> > > > >
> > > > >1. While the docker image outlined in KIP-975 is designed for
> > > > >production environments, it is equally suitable for development
> and
> > > > testing
> > > > >purposes. We will furnish the docker image, allowing users the
> > > > flexibility
> > > > >to employ it according to their specific needs.
> > > > >2. The configs will be injected into the docker container
> through
> > > > >environment variables. These environment variables wi

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Mickael Maison]

Hi Krishna,

Overall I'm supportive of having an official docker image.
I have a few questions:
- Can you clarify the process of selecting the Java version? Is the
proposal to only pick LTS versions? or to pick the highest version
supported by Kafka?
- Once a new Kafka version is released, what happens to the image
containing the previous release? Do we expect to still update it in
case of CVEs? If so for how long?
- How will we get notified that the base image has a CVE?
- Rather than having scripts PMC members have to run from their
machines, would it e possible to have a Jenkins job or GitHub action?

Thanks,
Mickael



On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma
 wrote:
>
> Hi Manikumar,
>
> Thanks for the feedback!
>
> 1. We propose the addition of a new directory named "docker" at the root of
> the repository, where all Docker-related code will be stored. A detailed
> directory structure has been added in the KIP.
> 2. We request the creation of an Apache Kafka repository (apache/kafka) on
> DockerHub, to be administered under the The Apache Software Foundation
> . The PMC members should have the
> necessary permissions for pushing updates to the docker repo.
>
> Thanks and regards,
> Vedarth
>
>
> On Fri, Oct 20, 2023 at 2:44 PM Manikumar  wrote:
>
> > Hi Krishna, Vedarth,
> >
> > Thanks for the KIP.
> >
> > 1. Can we add directory structure of Docker Image related files in Kafka
> > repo.
> >
> > 2. > Steps for the Docker image release will be included in the Release
> > Process doc of Apache Kafka
> >
> > Can we list down the requirements (repos, accounts) for releasing images to
> > docker hub. I am mainly asking because PMC needs to request docker hub
> > access/repos.
> > I can help in getting required repos/accounts.
> > https://infra.apache.org/docker-hub-policy.html
> >
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> > > Hi Viktor,
> > >
> > > I've noticed there are two types of custom jar configurations:
> > >
> > >1. *Type 1*: In this case, only the class name is required(e.g
> > > *authorizer.class.name
> > >**)* This can be configured by the
> > >following steps:
> > >   - Mount the jar in the container.
> > >   - Configure the *CLASSPATH* environment variable (used by
> > >   *kafka-run-class.sh*) by providing the mounted path to it. This can
> > >   be passed as an environment variable to the docker container.
> > >2. *Type 2*: Here, in addition to the class name, classpath can also
> > be
> > >configured (eg *remote.log.metadata.manager.class.name
> > > *and
> > >*remote.log.metadata.manager.class.path*). This can be configured by
> > the
> > >following steps:
> > >   - Mount the jar in the container.
> > >   - Configure the respective *class.path* property.
> > >
> > > Regards,
> > > Krishna
> > >
> > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > > > Hi Viktor,
> > > > Thanks for the questions.
> > > >
> > > >1. While the docker image outlined in KIP-975 is designed for
> > > >production environments, it is equally suitable for development and
> > > testing
> > > >purposes. We will furnish the docker image, allowing users the
> > > flexibility
> > > >to employ it according to their specific needs.
> > > >2. The configs will be injected into the docker container through
> > > >environment variables. These environment variables will have a
> > prefix
> > > >allowing for efficient parsing to extract the relevant
> > > properties.(Will add
> > > >this implementation in the KIP as well once we converge on this.)
> > > >3. Regarding this question, I'll conduct a test on my end after
> > > >gaining a better understanding, and then provide you with a
> > response.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > >
> > > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
> > > >  wrote:
> > > >
> > > >> Hi Ismael,
> > > >>
> > > >> I'm not trying to advocate against the docker image, I just pointed
> > out
> > > >> that the current scoping of the KIP may be a bit too generic and
> > thought
> > > >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can
> > > be
> > > >> discussed under one umbrella. Apologies if this was rooted in a
> > > >> misunderstanding.
> > > >>
> > > >> Kirshna,
> > > >>
> > > >> I think we need to refine the KIP a bit more. I think there are some
> > > >> interfaces that we need to include in the KIP as Kafka has plugins in
> > > >> certain cases where users are expected to provide implementation and I
> > > >> think it's worth discussing this in the KIP as they're kind of
> > > interfaces
> > > >> for users. Here are my questions in order:
> > > >> 1. In what environments do you want the ima

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hi Manikumar,

Thanks for the feedback!

1. We propose the addition of a new directory named "docker" at the root of
the repository, where all Docker-related code will be stored. A detailed
directory structure has been added in the KIP.
2. We request the creation of an Apache Kafka repository (apache/kafka) on
DockerHub, to be administered under the The Apache Software Foundation
. The PMC members should have the
necessary permissions for pushing updates to the docker repo.

Thanks and regards,
Vedarth


On Fri, Oct 20, 2023 at 2:44 PM Manikumar  wrote:

> Hi Krishna, Vedarth,
>
> Thanks for the KIP.
>
> 1. Can we add directory structure of Docker Image related files in Kafka
> repo.
>
> 2. > Steps for the Docker image release will be included in the Release
> Process doc of Apache Kafka
>
> Can we list down the requirements (repos, accounts) for releasing images to
> docker hub. I am mainly asking because PMC needs to request docker hub
> access/repos.
> I can help in getting required repos/accounts.
> https://infra.apache.org/docker-hub-policy.html
>
>
> Thanks,
> Manikumar
>
> On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Viktor,
> >
> > I've noticed there are two types of custom jar configurations:
> >
> >1. *Type 1*: In this case, only the class name is required(e.g
> > *authorizer.class.name
> >**)* This can be configured by the
> >following steps:
> >   - Mount the jar in the container.
> >   - Configure the *CLASSPATH* environment variable (used by
> >   *kafka-run-class.sh*) by providing the mounted path to it. This can
> >   be passed as an environment variable to the docker container.
> >2. *Type 2*: Here, in addition to the class name, classpath can also
> be
> >configured (eg *remote.log.metadata.manager.class.name
> > *and
> >*remote.log.metadata.manager.class.path*). This can be configured by
> the
> >following steps:
> >   - Mount the jar in the container.
> >   - Configure the respective *class.path* property.
> >
> > Regards,
> > Krishna
> >
> > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> > > Hi Viktor,
> > > Thanks for the questions.
> > >
> > >1. While the docker image outlined in KIP-975 is designed for
> > >production environments, it is equally suitable for development and
> > testing
> > >purposes. We will furnish the docker image, allowing users the
> > flexibility
> > >to employ it according to their specific needs.
> > >2. The configs will be injected into the docker container through
> > >environment variables. These environment variables will have a
> prefix
> > >allowing for efficient parsing to extract the relevant
> > properties.(Will add
> > >this implementation in the KIP as well once we converge on this.)
> > >3. Regarding this question, I'll conduct a test on my end after
> > >gaining a better understanding, and then provide you with a
> response.
> > >
> > > Regards,
> > > Krishna
> > >
> > >
> > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > >> Hi Ismael,
> > >>
> > >> I'm not trying to advocate against the docker image, I just pointed
> out
> > >> that the current scoping of the KIP may be a bit too generic and
> thought
> > >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can
> > be
> > >> discussed under one umbrella. Apologies if this was rooted in a
> > >> misunderstanding.
> > >>
> > >> Kirshna,
> > >>
> > >> I think we need to refine the KIP a bit more. I think there are some
> > >> interfaces that we need to include in the KIP as Kafka has plugins in
> > >> certain cases where users are expected to provide implementation and I
> > >> think it's worth discussing this in the KIP as they're kind of
> > interfaces
> > >> for users. Here are my questions in order:
> > >> 1. In what environments do you want the image to be used? As I
> > understand
> > >> it would replace the current testing image and serve as a basis for
> > >> development, but would it aim at production use cases too
> > (docker-compose,
> > >> Kubernetes, etc.)?
> > >> 2. How do you plan to forward configs to the broker? Do we expect a
> > >> populated server.properties file placed in a certain location or
> should
> > >> the
> > >> docker image create this file based on some input (like env vars)?
> > >> 3. Certain parts can be pluggable, like metric reporters or remote log
> > >> implementations that were just introduced by KIP-405. These manifest
> in
> > >> jar
> > >> files that must be put on the classpath of Kafka while certain
> > classnames
> > >> have to be configured. How do you plan to implement this, how do we
> > >> allow users to configure such things?
> > >>
> > >> Thanks,
> > >> Viktor
> > >>
> > >>
> > >>
> > >>
> > >> On T

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Manikumar]

Hi Krishna, Vedarth,

Thanks for the KIP.

1. Can we add directory structure of Docker Image related files in Kafka
repo.

2. > Steps for the Docker image release will be included in the Release
Process doc of Apache Kafka

Can we list down the requirements (repos, accounts) for releasing images to
docker hub. I am mainly asking because PMC needs to request docker hub
access/repos.
I can help in getting required repos/accounts.
https://infra.apache.org/docker-hub-policy.html


Thanks,
Manikumar

On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Viktor,
>
> I've noticed there are two types of custom jar configurations:
>
>1. *Type 1*: In this case, only the class name is required(e.g
> *authorizer.class.name
>**)* This can be configured by the
>following steps:
>   - Mount the jar in the container.
>   - Configure the *CLASSPATH* environment variable (used by
>   *kafka-run-class.sh*) by providing the mounted path to it. This can
>   be passed as an environment variable to the docker container.
>2. *Type 2*: Here, in addition to the class name, classpath can also be
>configured (eg *remote.log.metadata.manager.class.name
> *and
>*remote.log.metadata.manager.class.path*). This can be configured by the
>following steps:
>   - Mount the jar in the container.
>   - Configure the respective *class.path* property.
>
> Regards,
> Krishna
>
> On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Viktor,
> > Thanks for the questions.
> >
> >1. While the docker image outlined in KIP-975 is designed for
> >production environments, it is equally suitable for development and
> testing
> >purposes. We will furnish the docker image, allowing users the
> flexibility
> >to employ it according to their specific needs.
> >2. The configs will be injected into the docker container through
> >environment variables. These environment variables will have a prefix
> >allowing for efficient parsing to extract the relevant
> properties.(Will add
> >this implementation in the KIP as well once we converge on this.)
> >3. Regarding this question, I'll conduct a test on my end after
> >gaining a better understanding, and then provide you with a response.
> >
> > Regards,
> > Krishna
> >
> >
> > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
> >  wrote:
> >
> >> Hi Ismael,
> >>
> >> I'm not trying to advocate against the docker image, I just pointed out
> >> that the current scoping of the KIP may be a bit too generic and thought
> >> that KIP-974 and KIP-975 were aiming for mostly the same thing and can
> be
> >> discussed under one umbrella. Apologies if this was rooted in a
> >> misunderstanding.
> >>
> >> Kirshna,
> >>
> >> I think we need to refine the KIP a bit more. I think there are some
> >> interfaces that we need to include in the KIP as Kafka has plugins in
> >> certain cases where users are expected to provide implementation and I
> >> think it's worth discussing this in the KIP as they're kind of
> interfaces
> >> for users. Here are my questions in order:
> >> 1. In what environments do you want the image to be used? As I
> understand
> >> it would replace the current testing image and serve as a basis for
> >> development, but would it aim at production use cases too
> (docker-compose,
> >> Kubernetes, etc.)?
> >> 2. How do you plan to forward configs to the broker? Do we expect a
> >> populated server.properties file placed in a certain location or should
> >> the
> >> docker image create this file based on some input (like env vars)?
> >> 3. Certain parts can be pluggable, like metric reporters or remote log
> >> implementations that were just introduced by KIP-405. These manifest in
> >> jar
> >> files that must be put on the classpath of Kafka while certain
> classnames
> >> have to be configured. How do you plan to implement this, how do we
> >> allow users to configure such things?
> >>
> >> Thanks,
> >> Viktor
> >>
> >>
> >>
> >>
> >> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
> >>  wrote:
> >>
> >> > Hello,
> >> >
> >> > I think this would be a wonderful improvement to the ecosystem. While
> >> > Viktor is correct that most Docker pipelines eventually lead to a
> >> > kubernetes deployment, that should not stop us from creating an
> >> > Official Docker Image. Creating a Docker image would allow us to
> ensure
> >> a
> >> > level of quality and support for people who want to deploy Kafka as a
> >> > container on baremetal machines, it could allow us to create
> >> > a sandbox/developer environment for new contributors and developers to
> >> test
> >> > and have a single agreed upon environment that kafka works in for
> future
> >> > KIPs and would most likely spawn more contributions from people
> wanting
> >> to
> >> > optimize kafka for k8s.
> >>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Krishna Agarwal]

Hi Viktor,

I've noticed there are two types of custom jar configurations:

   1. *Type 1*: In this case, only the class name is required(e.g
*authorizer.class.name
   **)* This can be configured by the
   following steps:
  - Mount the jar in the container.
  - Configure the *CLASSPATH* environment variable (used by
  *kafka-run-class.sh*) by providing the mounted path to it. This can
  be passed as an environment variable to the docker container.
   2. *Type 2*: Here, in addition to the class name, classpath can also be
   configured (eg *remote.log.metadata.manager.class.name
    *and
   *remote.log.metadata.manager.class.path*). This can be configured by the
   following steps:
  - Mount the jar in the container.
  - Configure the respective *class.path* property.

Regards,
Krishna

On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Viktor,
> Thanks for the questions.
>
>1. While the docker image outlined in KIP-975 is designed for
>production environments, it is equally suitable for development and testing
>purposes. We will furnish the docker image, allowing users the flexibility
>to employ it according to their specific needs.
>2. The configs will be injected into the docker container through
>environment variables. These environment variables will have a prefix
>allowing for efficient parsing to extract the relevant properties.(Will add
>this implementation in the KIP as well once we converge on this.)
>3. Regarding this question, I'll conduct a test on my end after
>gaining a better understanding, and then provide you with a response.
>
> Regards,
> Krishna
>
>
> On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
>  wrote:
>
>> Hi Ismael,
>>
>> I'm not trying to advocate against the docker image, I just pointed out
>> that the current scoping of the KIP may be a bit too generic and thought
>> that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
>> discussed under one umbrella. Apologies if this was rooted in a
>> misunderstanding.
>>
>> Kirshna,
>>
>> I think we need to refine the KIP a bit more. I think there are some
>> interfaces that we need to include in the KIP as Kafka has plugins in
>> certain cases where users are expected to provide implementation and I
>> think it's worth discussing this in the KIP as they're kind of interfaces
>> for users. Here are my questions in order:
>> 1. In what environments do you want the image to be used? As I understand
>> it would replace the current testing image and serve as a basis for
>> development, but would it aim at production use cases too (docker-compose,
>> Kubernetes, etc.)?
>> 2. How do you plan to forward configs to the broker? Do we expect a
>> populated server.properties file placed in a certain location or should
>> the
>> docker image create this file based on some input (like env vars)?
>> 3. Certain parts can be pluggable, like metric reporters or remote log
>> implementations that were just introduced by KIP-405. These manifest in
>> jar
>> files that must be put on the classpath of Kafka while certain classnames
>> have to be configured. How do you plan to implement this, how do we
>> allow users to configure such things?
>>
>> Thanks,
>> Viktor
>>
>>
>>
>>
>> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
>>  wrote:
>>
>> > Hello,
>> >
>> > I think this would be a wonderful improvement to the ecosystem. While
>> > Viktor is correct that most Docker pipelines eventually lead to a
>> > kubernetes deployment, that should not stop us from creating an
>> > Official Docker Image. Creating a Docker image would allow us to ensure
>> a
>> > level of quality and support for people who want to deploy Kafka as a
>> > container on baremetal machines, it could allow us to create
>> > a sandbox/developer environment for new contributors and developers to
>> test
>> > and have a single agreed upon environment that kafka works in for future
>> > KIPs and would most likely spawn more contributions from people wanting
>> to
>> > optimize kafka for k8s.
>> >
>> >
>> > I am 100% for this and will gladly help if approved.
>> >
>> > Kenneth
>> >
>> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
>> >
>> > > Hi Viktor,
>> > >
>> > > I disagree. Docker is a very popular deployment tool and it's not only
>> > used
>> > > with Kubernetes.
>> > >
>> > > Ismael
>> > >
>> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
>> > >  wrote:
>> > >
>> > > > Hi Krishna,
>> > > >
>> > > > I think you should merge this KIP and KIP-974
>> >  as there
>> are
>> > overlaps as
>> > > > Federico pointed out on KIP-974
>> > . I think
>> you
>> > should keep that one as it
>> > > > has well defined goals (improve tests) while I feel this one is too
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hey folks!

We've incorporated all the points discussed into the KIP. Additionally,
we've included a section detailing the Release Process of the Docker image.
We value your feedback, so please take some time to review it. Once we have
your inputs, we can proceed with the voting process.

Thanks and regards,
Vedarth


On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal 
wrote:

>
> > Hi,
> > Apache Kafka does not have an official docker image currently.
> > I want to submit a KIP to publish a docker image for Apache Kafka.
> >
> > KIP-975: Docker Image for Apache Kafka
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > >
> >
> > Regards,
> > Krishna
> >
>

On Thu, Sep 14, 2023 at 1:44 PM Viktor Somogyi-Vass
 wrote:

> Hi Krishna,
>
> I think you should merge this KIP and KIP-974 as there are overlaps as
> Federico pointed out on KIP-974. I think you should keep that one as it
> has well defined goals (improve tests) while I feel this one is too
> generic. Docker is usually just a tool for either testing or Kubernetes, so
> they have very well defined use-cases. In the case of Flink for instance
> the image is used for its kubernetes operator. The use case would determine
> a lot of things and I think a generic image would likely not fit the needs
> of all use-cases.
>
> Best,
> Viktor
>
> On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> krishna0608agar...@gmail.com>
> wrote:
>
> > Hi,
> > Apache Kafka does not have an official docker image currently.
> > I want to submit a KIP to publish a docker image for Apache Kafka.
> >
> > KIP-975: Docker Image for Apache Kafka
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > >
> >
> > Regards,
> > Krishna
> >
>

[DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hey folks!

Please feel free to ask any additional questions concerning this KIP.
If there are no additional queries, shall we consider moving forward with
the voting process?

Thanks and regards,
Vedarth

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Vedarth Sharma]

Hey Viktor! Thanks for bringing up this use case.
I think we can take advantage of Docker volume for this.
We can allow users to mount a folder containing the secret files.
This folder can then be used to pass secrets to the container.

Thanks and regards,
Vedarth

On Wed, Sep 27, 2023 at 4:39 PM Viktor Somogyi-Vass
 wrote:

> Hi Krishna,
>
> Thanks for the answer. I've seen multiple such images where environment
> variables are used and I think they are generally good but it's unsafe for
> passing around secrets, jaas configs and so on. Perhaps for secrets we
> could recommend using the file config provider. Then users can create and
> mount secured properties file(s) with configs that are considered as
> secrets. What do you think? Did you already have something in your mind
> regarding this?
>
> Thanks,
> Viktor
>
> On Tue, Sep 26, 2023 at 3:05 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Ismael,
> > Apologies for missing the mailing list in the last reply.
> >
> > Thank you for the suggestions.
> > Just to clarify, the sizes mentioned in the previous email are of the
> > uncompressed base images, not the resulting Apache Kafka docker images:
> >
> >1. eclipse-temurin:17-jre -- 263MB (They should release JRE 21 images
> >soon)
> >2.
> registry.access.redhat.com/ubi8/openjdk-17-runtime:1.17-1.1693366274
> >-- 375MB
> >
> > Regards,
> > Krishna
> >
> >
> > On Tue, Sep 26, 2023 at 9:24 AM Ismael Juma  wrote:
> >
> > > Hi Krishna,
> > >
> > > Looks like you sent the response to me and not the mailing list,
> > > please include the mailing list in the replies. Comments below.
> > >
> > > On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > >> Hi Ismael,
> > >> Thanks for the questions.
> > >>
> > >>1. We intend to support only the latest Java supported by Apache
> > >>Kafka(As per this documentation
> > >> Apache Kafka
> > currently
> > >>supports Java 8, Java 11, and Java 17) which currently is Java 17.
> If
> > >>Apache Kafka supports Java 21 in the future, we will align with it.
> > >>
> > >> We are already building and testing with Java 21 (
> > > https://github.com/apache/kafka/pull/14451 updates `README.md` to
> > > indicate that). By 3.7.0 (the next release), we'll have Java 21 as one
> of
> > > the officially supported versions. I think we should start with that
> > > version for both docker image KIPs.
> > >
> > >>
> > >>1. For users seeking a Docker image with an alternative Java
> version,
> > >>they will have the flexibility to build their own Docker image
> > utilising
> > >>the Dockerfiles we provide. In our documentation, we will provide
> > clear
> > >>guidance on the designated base images for various Java versions.
> > >>
> > >> This sounds good to me. We should include these details as part of the
> > > KIP and also the documentation for the docker images. More
> specifically,
> > we
> > > should state that we will update the Java major version as part of
> minor
> > > Apache Kafka releases. The implication is that users who include broker
> > > plugins alongside the broker should use custom images to ensure their
> > > custom code is not broken by Java upgrades.
> > >
> > >>
> > >>1. Apache Kafka only requires JRE, not JDK, for operation.
> Utilizing
> > >>a base image with only JRE, rather than JDK, is a logical choice as
> > it
> > >>significantly reduces the size of the docker image.
> > >>Upon further investigation, I discovered the eclipse-temurin
> > >><
> >
> https://hub.docker.com/layers/library/eclipse-temurin/17-jre/images/sha256-d1dfb065ae433fe1b43ac7e50a1ed03660f487c73ec256c686b126c37fd4d086?context=explore
> > >
> > >>docker image, which is notably smaller than Redhat’s ubi8 docker
> > image (263
> > >>MB vs 375 MB). Additionally, the fact that Apache Flink relies on
> > >>eclipse-temurin base images
> > >><
> >
> https://github.com/apache/flink-docker/blob/master/1.17/scala_2.12-java11-ubuntu/Dockerfile#L19
> > >
> > >>further increases our confidence in their dependability(Will make
> > this
> > >>change in the KIP).
> > >>
> > >> Yes, eclipse-temurin looks like a good choice to me. Nice size
> > reduction!
> > >
> > >>
> > >>1. I'll conduct comparisons between our docker image and existing
> > >>ones, and incorporate the findings into the KIP. I'll keep you
> > posted on
> > >>the same.
> > >>
> > >> Excellent, thanks!
> > >
> > > Ismael
> > >
> > >
> > >> On Wed, Sep 20, 2023 at 11:26 PM Ismael Juma 
> wrote:
> > >>
> > >>> Hi Krishna,
> > >>>
> > >>> Thanks for the KIP. A few quick questions:
> > >>>
> > >>> 1. Since this will only be available for Kafka 3.7 in the best case,
> I
> > >>> suggest we go with Java 21 instead of Java 17. Also, we should be
> clear
> > >>> about Java version expectations. Are we allowed to change the Java
> > >>> vers

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Viktor Somogyi-Vass]

Hi Krishna,

Thanks for the answer. I've seen multiple such images where environment
variables are used and I think they are generally good but it's unsafe for
passing around secrets, jaas configs and so on. Perhaps for secrets we
could recommend using the file config provider. Then users can create and
mount secured properties file(s) with configs that are considered as
secrets. What do you think? Did you already have something in your mind
regarding this?

Thanks,
Viktor

On Tue, Sep 26, 2023 at 3:05 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Ismael,
> Apologies for missing the mailing list in the last reply.
>
> Thank you for the suggestions.
> Just to clarify, the sizes mentioned in the previous email are of the
> uncompressed base images, not the resulting Apache Kafka docker images:
>
>1. eclipse-temurin:17-jre -- 263MB (They should release JRE 21 images
>soon)
>2. registry.access.redhat.com/ubi8/openjdk-17-runtime:1.17-1.1693366274
>-- 375MB
>
> Regards,
> Krishna
>
>
> On Tue, Sep 26, 2023 at 9:24 AM Ismael Juma  wrote:
>
> > Hi Krishna,
> >
> > Looks like you sent the response to me and not the mailing list,
> > please include the mailing list in the replies. Comments below.
> >
> > On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> >> Hi Ismael,
> >> Thanks for the questions.
> >>
> >>1. We intend to support only the latest Java supported by Apache
> >>Kafka(As per this documentation
> >> Apache Kafka
> currently
> >>supports Java 8, Java 11, and Java 17) which currently is Java 17. If
> >>Apache Kafka supports Java 21 in the future, we will align with it.
> >>
> >> We are already building and testing with Java 21 (
> > https://github.com/apache/kafka/pull/14451 updates `README.md` to
> > indicate that). By 3.7.0 (the next release), we'll have Java 21 as one of
> > the officially supported versions. I think we should start with that
> > version for both docker image KIPs.
> >
> >>
> >>1. For users seeking a Docker image with an alternative Java version,
> >>they will have the flexibility to build their own Docker image
> utilising
> >>the Dockerfiles we provide. In our documentation, we will provide
> clear
> >>guidance on the designated base images for various Java versions.
> >>
> >> This sounds good to me. We should include these details as part of the
> > KIP and also the documentation for the docker images. More specifically,
> we
> > should state that we will update the Java major version as part of minor
> > Apache Kafka releases. The implication is that users who include broker
> > plugins alongside the broker should use custom images to ensure their
> > custom code is not broken by Java upgrades.
> >
> >>
> >>1. Apache Kafka only requires JRE, not JDK, for operation. Utilizing
> >>a base image with only JRE, rather than JDK, is a logical choice as
> it
> >>significantly reduces the size of the docker image.
> >>Upon further investigation, I discovered the eclipse-temurin
> >><
> https://hub.docker.com/layers/library/eclipse-temurin/17-jre/images/sha256-d1dfb065ae433fe1b43ac7e50a1ed03660f487c73ec256c686b126c37fd4d086?context=explore
> >
> >>docker image, which is notably smaller than Redhat’s ubi8 docker
> image (263
> >>MB vs 375 MB). Additionally, the fact that Apache Flink relies on
> >>eclipse-temurin base images
> >><
> https://github.com/apache/flink-docker/blob/master/1.17/scala_2.12-java11-ubuntu/Dockerfile#L19
> >
> >>further increases our confidence in their dependability(Will make
> this
> >>change in the KIP).
> >>
> >> Yes, eclipse-temurin looks like a good choice to me. Nice size
> reduction!
> >
> >>
> >>1. I'll conduct comparisons between our docker image and existing
> >>ones, and incorporate the findings into the KIP. I'll keep you
> posted on
> >>the same.
> >>
> >> Excellent, thanks!
> >
> > Ismael
> >
> >
> >> On Wed, Sep 20, 2023 at 11:26 PM Ismael Juma  wrote:
> >>
> >>> Hi Krishna,
> >>>
> >>> Thanks for the KIP. A few quick questions:
> >>>
> >>> 1. Since this will only be available for Kafka 3.7 in the best case, I
> >>> suggest we go with Java 21 instead of Java 17. Also, we should be clear
> >>> about Java version expectations. Are we allowed to change the Java
> >>> version
> >>> whenever we want? Or can it only be done in major versions? I prefer
> the
> >>> former option by default, but a subset of users may have their own
> >>> plugins
> >>> running within the broker. Such users may need to have control over the
> >>> Java version they use - how would they achieve tha
> >>> 2. We state that ubi-minimal would be the base image - it would be good
> >>> to
> >>> provide some details on why this was the choice.
> >>> 3. How does this docker image compare to the existing popular images?
> >>> Perhaps we can compare with the 3 most popular image

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Krishna Agarwal]

Hi Ismael,
Apologies for missing the mailing list in the last reply.

Thank you for the suggestions.
Just to clarify, the sizes mentioned in the previous email are of the
uncompressed base images, not the resulting Apache Kafka docker images:

   1. eclipse-temurin:17-jre -- 263MB (They should release JRE 21 images
   soon)
   2. registry.access.redhat.com/ubi8/openjdk-17-runtime:1.17-1.1693366274
   -- 375MB

Regards,
Krishna


On Tue, Sep 26, 2023 at 9:24 AM Ismael Juma  wrote:

> Hi Krishna,
>
> Looks like you sent the response to me and not the mailing list,
> please include the mailing list in the replies. Comments below.
>
> On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
>> Hi Ismael,
>> Thanks for the questions.
>>
>>1. We intend to support only the latest Java supported by Apache
>>Kafka(As per this documentation
>> Apache Kafka currently
>>supports Java 8, Java 11, and Java 17) which currently is Java 17. If
>>Apache Kafka supports Java 21 in the future, we will align with it.
>>
>> We are already building and testing with Java 21 (
> https://github.com/apache/kafka/pull/14451 updates `README.md` to
> indicate that). By 3.7.0 (the next release), we'll have Java 21 as one of
> the officially supported versions. I think we should start with that
> version for both docker image KIPs.
>
>>
>>1. For users seeking a Docker image with an alternative Java version,
>>they will have the flexibility to build their own Docker image utilising
>>the Dockerfiles we provide. In our documentation, we will provide clear
>>guidance on the designated base images for various Java versions.
>>
>> This sounds good to me. We should include these details as part of the
> KIP and also the documentation for the docker images. More specifically, we
> should state that we will update the Java major version as part of minor
> Apache Kafka releases. The implication is that users who include broker
> plugins alongside the broker should use custom images to ensure their
> custom code is not broken by Java upgrades.
>
>>
>>1. Apache Kafka only requires JRE, not JDK, for operation. Utilizing
>>a base image with only JRE, rather than JDK, is a logical choice as it
>>significantly reduces the size of the docker image.
>>Upon further investigation, I discovered the eclipse-temurin
>>
>> 
>>docker image, which is notably smaller than Redhat’s ubi8 docker image 
>> (263
>>MB vs 375 MB). Additionally, the fact that Apache Flink relies on
>>eclipse-temurin base images
>>
>> 
>>further increases our confidence in their dependability(Will make this
>>change in the KIP).
>>
>> Yes, eclipse-temurin looks like a good choice to me. Nice size reduction!
>
>>
>>1. I'll conduct comparisons between our docker image and existing
>>ones, and incorporate the findings into the KIP. I'll keep you posted on
>>the same.
>>
>> Excellent, thanks!
>
> Ismael
>
>
>> On Wed, Sep 20, 2023 at 11:26 PM Ismael Juma  wrote:
>>
>>> Hi Krishna,
>>>
>>> Thanks for the KIP. A few quick questions:
>>>
>>> 1. Since this will only be available for Kafka 3.7 in the best case, I
>>> suggest we go with Java 21 instead of Java 17. Also, we should be clear
>>> about Java version expectations. Are we allowed to change the Java
>>> version
>>> whenever we want? Or can it only be done in major versions? I prefer the
>>> former option by default, but a subset of users may have their own
>>> plugins
>>> running within the broker. Such users may need to have control over the
>>> Java version they use - how would they achieve tha
>>> 2. We state that ubi-minimal would be the base image - it would be good
>>> to
>>> provide some details on why this was the choice.
>>> 3. How does this docker image compare to the existing popular images?
>>> Perhaps we can compare with the 3 most popular images for Apache Kafka
>>> these days.
>>>
>>> Ismael
>>>
>>> On Fri, Sep 8, 2023 at 12:57 AM Krishna Agarwal <
>>> krishna0608agar...@gmail.com> wrote:
>>>
>>> > Hi,
>>> > Apache Kafka does not have an official docker image currently.
>>> > I want to submit a KIP to publish a docker image for Apache Kafka.
>>> >
>>> > KIP-975: Docker Image for Apache Kafka
>>> > <
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>>> > >
>>> >
>>> > Regards,
>>> > Krishna
>>> >
>>>
>>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Krishna,

Looks like you sent the response to me and not the mailing list,
please include the mailing list in the replies. Comments below.

On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Ismael,
> Thanks for the questions.
>
>1. We intend to support only the latest Java supported by Apache
>Kafka(As per this documentation
> Apache Kafka currently
>supports Java 8, Java 11, and Java 17) which currently is Java 17. If
>Apache Kafka supports Java 21 in the future, we will align with it.
>
> We are already building and testing with Java 21 (
https://github.com/apache/kafka/pull/14451 updates `README.md` to indicate
that). By 3.7.0 (the next release), we'll have Java 21 as one of the
officially supported versions. I think we should start with that version
for both docker image KIPs.

>
>1. For users seeking a Docker image with an alternative Java version,
>they will have the flexibility to build their own Docker image utilising
>the Dockerfiles we provide. In our documentation, we will provide clear
>guidance on the designated base images for various Java versions.
>
> This sounds good to me. We should include these details as part of the KIP
and also the documentation for the docker images. More specifically, we
should state that we will update the Java major version as part of minor
Apache Kafka releases. The implication is that users who include broker
plugins alongside the broker should use custom images to ensure their
custom code is not broken by Java upgrades.

>
>1. Apache Kafka only requires JRE, not JDK, for operation. Utilizing a
>base image with only JRE, rather than JDK, is a logical choice as it
>significantly reduces the size of the docker image.
>Upon further investigation, I discovered the eclipse-temurin
>
> 
>docker image, which is notably smaller than Redhat’s ubi8 docker image (263
>MB vs 375 MB). Additionally, the fact that Apache Flink relies on
>eclipse-temurin base images
>
> 
>further increases our confidence in their dependability(Will make this
>change in the KIP).
>
> Yes, eclipse-temurin looks like a good choice to me. Nice size reduction!

>
>1. I'll conduct comparisons between our docker image and existing
>ones, and incorporate the findings into the KIP. I'll keep you posted on
>the same.
>
> Excellent, thanks!

Ismael


> On Wed, Sep 20, 2023 at 11:26 PM Ismael Juma  wrote:
>
>> Hi Krishna,
>>
>> Thanks for the KIP. A few quick questions:
>>
>> 1. Since this will only be available for Kafka 3.7 in the best case, I
>> suggest we go with Java 21 instead of Java 17. Also, we should be clear
>> about Java version expectations. Are we allowed to change the Java version
>> whenever we want? Or can it only be done in major versions? I prefer the
>> former option by default, but a subset of users may have their own plugins
>> running within the broker. Such users may need to have control over the
>> Java version they use - how would they achieve tha
>> 2. We state that ubi-minimal would be the base image - it would be good to
>> provide some details on why this was the choice.
>> 3. How does this docker image compare to the existing popular images?
>> Perhaps we can compare with the 3 most popular images for Apache Kafka
>> these days.
>>
>> Ismael
>>
>> On Fri, Sep 8, 2023 at 12:57 AM Krishna Agarwal <
>> krishna0608agar...@gmail.com> wrote:
>>
>> > Hi,
>> > Apache Kafka does not have an official docker image currently.
>> > I want to submit a KIP to publish a docker image for Apache Kafka.
>> >
>> > KIP-975: Docker Image for Apache Kafka
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>> > >
>> >
>> > Regards,
>> > Krishna
>> >
>>
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Krishna Agarwal]

Hi Viktor,
Thanks for the questions.

   1. While the docker image outlined in KIP-975 is designed for production
   environments, it is equally suitable for development and testing purposes.
   We will furnish the docker image, allowing users the flexibility to employ
   it according to their specific needs.
   2. The configs will be injected into the docker container through
   environment variables. These environment variables will have a prefix
   allowing for efficient parsing to extract the relevant properties.(Will add
   this implementation in the KIP as well once we converge on this.)
   3. Regarding this question, I'll conduct a test on my end after gaining
   a better understanding, and then provide you with a response.

Regards,
Krishna


On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
 wrote:

> Hi Ismael,
>
> I'm not trying to advocate against the docker image, I just pointed out
> that the current scoping of the KIP may be a bit too generic and thought
> that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
> discussed under one umbrella. Apologies if this was rooted in a
> misunderstanding.
>
> Kirshna,
>
> I think we need to refine the KIP a bit more. I think there are some
> interfaces that we need to include in the KIP as Kafka has plugins in
> certain cases where users are expected to provide implementation and I
> think it's worth discussing this in the KIP as they're kind of interfaces
> for users. Here are my questions in order:
> 1. In what environments do you want the image to be used? As I understand
> it would replace the current testing image and serve as a basis for
> development, but would it aim at production use cases too (docker-compose,
> Kubernetes, etc.)?
> 2. How do you plan to forward configs to the broker? Do we expect a
> populated server.properties file placed in a certain location or should the
> docker image create this file based on some input (like env vars)?
> 3. Certain parts can be pluggable, like metric reporters or remote log
> implementations that were just introduced by KIP-405. These manifest in jar
> files that must be put on the classpath of Kafka while certain classnames
> have to be configured. How do you plan to implement this, how do we
> allow users to configure such things?
>
> Thanks,
> Viktor
>
>
>
>
> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
>  wrote:
>
> > Hello,
> >
> > I think this would be a wonderful improvement to the ecosystem. While
> > Viktor is correct that most Docker pipelines eventually lead to a
> > kubernetes deployment, that should not stop us from creating an
> > Official Docker Image. Creating a Docker image would allow us to ensure a
> > level of quality and support for people who want to deploy Kafka as a
> > container on baremetal machines, it could allow us to create
> > a sandbox/developer environment for new contributors and developers to
> test
> > and have a single agreed upon environment that kafka works in for future
> > KIPs and would most likely spawn more contributions from people wanting
> to
> > optimize kafka for k8s.
> >
> >
> > I am 100% for this and will gladly help if approved.
> >
> > Kenneth
> >
> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
> >
> > > Hi Viktor,
> > >
> > > I disagree. Docker is a very popular deployment tool and it's not only
> > used
> > > with Kubernetes.
> > >
> > > Ismael
> > >
> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > I think you should merge this KIP and KIP-974
> >  as there are
> > overlaps as
> > > > Federico pointed out on KIP-974
> > . I think you
> > should keep that one as it
> > > > has well defined goals (improve tests) while I feel this one is too
> > > > generic. Docker is usually just a tool for either testing or
> > Kubernetes,
> > > so
> > > > they have very well defined use-cases. In the case of Flink for
> > instance
> > > > the image is used for its kubernetes operator. The use case would
> > > determine
> > > > a lot of things and I think a generic image would likely not fit the
> > > needs
> > > > of all use-cases.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Apache Kafka does not have an official docker image currently.
> > > > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > > >
> > > > > KIP-975  >:
> > Docker Image for Apache Kafka
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > >
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Krishna,

Thanks for the KIP. A few quick questions:

1. Since this will only be available for Kafka 3.7 in the best case, I
suggest we go with Java 21 instead of Java 17. Also, we should be clear
about Java version expectations. Are we allowed to change the Java version
whenever we want? Or can it only be done in major versions? I prefer the
former option by default, but a subset of users may have their own plugins
running within the broker. Such users may need to have control over the
Java version they use - how would they achieve tha
2. We state that ubi-minimal would be the base image - it would be good to
provide some details on why this was the choice.
3. How does this docker image compare to the existing popular images?
Perhaps we can compare with the 3 most popular images for Apache Kafka
these days.

Ismael

On Fri, Sep 8, 2023 at 12:57 AM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> Apache Kafka does not have an official docker image currently.
> I want to submit a KIP to publish a docker image for Apache Kafka.
>
> KIP-975: Docker Image for Apache Kafka
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >
>
> Regards,
> Krishna
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Viktor,

Thanks for the clarification. This is a separate KIP because the scope is
simpler than the graalvm docker images - there is much more to figure out
when it comes to using Kafka with graalvm.

Ismael

On Tue, Sep 19, 2023 at 3:12 AM Viktor Somogyi-Vass
 wrote:

> Hi Ismael,
>
> I'm not trying to advocate against the docker image, I just pointed out
> that the current scoping of the KIP may be a bit too generic and thought
> that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
> discussed under one umbrella. Apologies if this was rooted in a
> misunderstanding.
>
> Kirshna,
>
> I think we need to refine the KIP a bit more. I think there are some
> interfaces that we need to include in the KIP as Kafka has plugins in
> certain cases where users are expected to provide implementation and I
> think it's worth discussing this in the KIP as they're kind of interfaces
> for users. Here are my questions in order:
> 1. In what environments do you want the image to be used? As I understand
> it would replace the current testing image and serve as a basis for
> development, but would it aim at production use cases too (docker-compose,
> Kubernetes, etc.)?
> 2. How do you plan to forward configs to the broker? Do we expect a
> populated server.properties file placed in a certain location or should the
> docker image create this file based on some input (like env vars)?
> 3. Certain parts can be pluggable, like metric reporters or remote log
> implementations that were just introduced by KIP-405. These manifest in jar
> files that must be put on the classpath of Kafka while certain classnames
> have to be configured. How do you plan to implement this, how do we
> allow users to configure such things?
>
> Thanks,
> Viktor
>
>
>
>
> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
>  wrote:
>
> > Hello,
> >
> > I think this would be a wonderful improvement to the ecosystem. While
> > Viktor is correct that most Docker pipelines eventually lead to a
> > kubernetes deployment, that should not stop us from creating an
> > Official Docker Image. Creating a Docker image would allow us to ensure a
> > level of quality and support for people who want to deploy Kafka as a
> > container on baremetal machines, it could allow us to create
> > a sandbox/developer environment for new contributors and developers to
> test
> > and have a single agreed upon environment that kafka works in for future
> > KIPs and would most likely spawn more contributions from people wanting
> to
> > optimize kafka for k8s.
> >
> >
> > I am 100% for this and will gladly help if approved.
> >
> > Kenneth
> >
> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
> >
> > > Hi Viktor,
> > >
> > > I disagree. Docker is a very popular deployment tool and it's not only
> > used
> > > with Kubernetes.
> > >
> > > Ismael
> > >
> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > I think you should merge this KIP and KIP-974
> >  as there are
> > overlaps as
> > > > Federico pointed out on KIP-974
> > . I think you
> > should keep that one as it
> > > > has well defined goals (improve tests) while I feel this one is too
> > > > generic. Docker is usually just a tool for either testing or
> > Kubernetes,
> > > so
> > > > they have very well defined use-cases. In the case of Flink for
> > instance
> > > > the image is used for its kubernetes operator. The use case would
> > > determine
> > > > a lot of things and I think a generic image would likely not fit the
> > > needs
> > > > of all use-cases.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Apache Kafka does not have an official docker image currently.
> > > > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > > >
> > > > > KIP-975  >:
> > Docker Image for Apache Kafka
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > >
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Viktor Somogyi-Vass]

Hi Ismael,

I'm not trying to advocate against the docker image, I just pointed out
that the current scoping of the KIP may be a bit too generic and thought
that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
discussed under one umbrella. Apologies if this was rooted in a
misunderstanding.

Kirshna,

I think we need to refine the KIP a bit more. I think there are some
interfaces that we need to include in the KIP as Kafka has plugins in
certain cases where users are expected to provide implementation and I
think it's worth discussing this in the KIP as they're kind of interfaces
for users. Here are my questions in order:
1. In what environments do you want the image to be used? As I understand
it would replace the current testing image and serve as a basis for
development, but would it aim at production use cases too (docker-compose,
Kubernetes, etc.)?
2. How do you plan to forward configs to the broker? Do we expect a
populated server.properties file placed in a certain location or should the
docker image create this file based on some input (like env vars)?
3. Certain parts can be pluggable, like metric reporters or remote log
implementations that were just introduced by KIP-405. These manifest in jar
files that must be put on the classpath of Kafka while certain classnames
have to be configured. How do you plan to implement this, how do we
allow users to configure such things?

Thanks,
Viktor




On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
 wrote:

> Hello,
>
> I think this would be a wonderful improvement to the ecosystem. While
> Viktor is correct that most Docker pipelines eventually lead to a
> kubernetes deployment, that should not stop us from creating an
> Official Docker Image. Creating a Docker image would allow us to ensure a
> level of quality and support for people who want to deploy Kafka as a
> container on baremetal machines, it could allow us to create
> a sandbox/developer environment for new contributors and developers to test
> and have a single agreed upon environment that kafka works in for future
> KIPs and would most likely spawn more contributions from people wanting to
> optimize kafka for k8s.
>
>
> I am 100% for this and will gladly help if approved.
>
> Kenneth
>
> On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
>
> > Hi Viktor,
> >
> > I disagree. Docker is a very popular deployment tool and it's not only
> used
> > with Kubernetes.
> >
> > Ismael
> >
> > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Hi Krishna,
> > >
> > > I think you should merge this KIP and KIP-974
>  as there are
> overlaps as
> > > Federico pointed out on KIP-974
> . I think you
> should keep that one as it
> > > has well defined goals (improve tests) while I feel this one is too
> > > generic. Docker is usually just a tool for either testing or
> Kubernetes,
> > so
> > > they have very well defined use-cases. In the case of Flink for
> instance
> > > the image is used for its kubernetes operator. The use case would
> > determine
> > > a lot of things and I think a generic image would likely not fit the
> > needs
> > > of all use-cases.
> > >
> > > Best,
> > > Viktor
> > >
> > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > > krishna0608agar...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > Apache Kafka does not have an official docker image currently.
> > > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > >
> > > > KIP-975 :
> Docker Image for Apache Kafka
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > >
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Kenneth Eversole]

Hello,

I think this would be a wonderful improvement to the ecosystem. While
Viktor is correct that most Docker pipelines eventually lead to a
kubernetes deployment, that should not stop us from creating an
Official Docker Image. Creating a Docker image would allow us to ensure a
level of quality and support for people who want to deploy Kafka as a
container on baremetal machines, it could allow us to create
a sandbox/developer environment for new contributors and developers to test
and have a single agreed upon environment that kafka works in for future
KIPs and would most likely spawn more contributions from people wanting to
optimize kafka for k8s.


I am 100% for this and will gladly help if approved.

Kenneth

On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:

> Hi Viktor,
>
> I disagree. Docker is a very popular deployment tool and it's not only used
> with Kubernetes.
>
> Ismael
>
> On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
>  wrote:
>
> > Hi Krishna,
> >
> > I think you should merge this KIP and KIP-974 as there are overlaps as
> > Federico pointed out on KIP-974. I think you should keep that one as it
> > has well defined goals (improve tests) while I feel this one is too
> > generic. Docker is usually just a tool for either testing or Kubernetes,
> so
> > they have very well defined use-cases. In the case of Flink for instance
> > the image is used for its kubernetes operator. The use case would
> determine
> > a lot of things and I think a generic image would likely not fit the
> needs
> > of all use-cases.
> >
> > Best,
> > Viktor
> >
> > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > krishna0608agar...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > Apache Kafka does not have an official docker image currently.
> > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > >
> > > KIP-975: Docker Image for Apache Kafka
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > >
> > >
> > > Regards,
> > > Krishna
> > >
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Viktor,

I disagree. Docker is a very popular deployment tool and it's not only used
with Kubernetes.

Ismael

On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
 wrote:

> Hi Krishna,
>
> I think you should merge this KIP and KIP-974 as there are overlaps as
> Federico pointed out on KIP-974. I think you should keep that one as it
> has well defined goals (improve tests) while I feel this one is too
> generic. Docker is usually just a tool for either testing or Kubernetes, so
> they have very well defined use-cases. In the case of Flink for instance
> the image is used for its kubernetes operator. The use case would determine
> a lot of things and I think a generic image would likely not fit the needs
> of all use-cases.
>
> Best,
> Viktor
>
> On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> krishna0608agar...@gmail.com>
> wrote:
>
> > Hi,
> > Apache Kafka does not have an official docker image currently.
> > I want to submit a KIP to publish a docker image for Apache Kafka.
> >
> > KIP-975: Docker Image for Apache Kafka
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > >
> >
> > Regards,
> > Krishna
> >
>

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Viktor Somogyi-Vass]

Hi Krishna,

I think you should merge this KIP and KIP-974 as there are overlaps as
Federico pointed out on KIP-974. I think you should keep that one as it
has well defined goals (improve tests) while I feel this one is too
generic. Docker is usually just a tool for either testing or Kubernetes, so
they have very well defined use-cases. In the case of Flink for instance
the image is used for its kubernetes operator. The use case would determine
a lot of things and I think a generic image would likely not fit the needs
of all use-cases.

Best,
Viktor

On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal 
wrote:

> Hi,
> Apache Kafka does not have an official docker image currently.
> I want to submit a KIP to publish a docker image for Apache Kafka.
>
> KIP-975: Docker Image for Apache Kafka
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >
>
> Regards,
> Krishna
>

[DISCUSS] KIP-975 Docker Image for Apache Kafka

[Krishna Agarwal]

Hi,
Apache Kafka does not have an official docker image currently.
I want to submit a KIP to publish a docker image for Apache Kafka.

KIP-975: Docker Image for Apache Kafka


Regards,
Krishna