Mickael, If reload4j is fully compatible, then no vote should be needed. It would be great if a committer could review the PR, test it and verify that this is the case. For example, I noticed that the maven coordinates are not the same. What happens if both log4j and reload4j are in the classpath? Is this an issue? Similarly "slf4j-reload4j" seems to be a different artifact. I left some comments in the PR too.
Ismael On Thu, Mar 24, 2022 at 7:08 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi, > > So to summarize, it seems the plan is to: > - adopt reload4j in Kafka 3.2.0 > - delay the switch to log4j2 to Kafka 4.0.0 > > Reload4j is supposed to be a fully compatible drop-in replacement for > log4j. Do we still want to do a vote for the switch? It looks like > there's already a JIRA and a PR ready: > https://issues.apache.org/jira/browse/KAFKA-13660 > https://github.com/apache/kafka/pull/11743 > > Do we also want to switch for 3.1.1? > > Adding Bruno and Tom on CC as they are the release managers for 3.2.0 and > 3.1.1. > > Thanks, > Mickael > > On Thu, Mar 24, 2022 at 7:06 AM Dongjin Lee <dong...@apache.org> wrote: > > > > No. It is why I have no firm position. I will follow the community's > > decision. > > > > Thanks, > > Dongjin > > > > On Thu, Mar 24, 2022 at 12:34 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi Dongjin, > > > > > > We really appreciate the super valuable work you've been doing here. > Do we > > > have evidence that customers don't use custom filters/layouts? > > > > > > Ismael > > > > > > On Wed, Mar 23, 2022 at 7:53 AM Dongjin Lee <dong...@apache.org> > wrote: > > > > > > > Hi Mikael, Edoardo and Ismael, > > > > > > > > Sorry for being late. Frankly, I thought KIP-653 is not a breaking > change > > > > since (as Edoardo stated) unless the user uses custom filters or > layouts, > > > > log4j-1.2-api.jar 'bridge' jar can handle the cases. It is why the > > > > 'Compatibility, Deprecation, and Migration Plan' section of the > document > > > is > > > > so brief. (As far as I know, it is such a rare case, and I thought it > > > would > > > > not be so problematic.) > > > > > > > > I have no firm position on the release plan of this feature. > Regardless > > > of > > > > whether the community decides to put off the adoption of log4j2 to > 4.0, I > > > > will maintain the PR and the preview releases up-to-date as far as > > > possible > > > > - although it can be significantly late for my main job. The > decision is > > > > totally up to the community - No matter how it is decided, I will > follow. > > > > > > > > Best, > > > > Dongjin > > > > > > > > p.s. @Edoardo I'm HIM. (HAHA) > > > > > > > > On Wed, Mar 23, 2022 at 10:12 PM Ismael Juma <ism...@juma.me.uk> > wrote: > > > > > > > > > Hi Mickael, > > > > > > > > > > Thanks for your feedback. I agree with the importance of fixing the > > > CVEs > > > > > and also of not breaking compatibility in a critical layer. > Regarding > > > > > Apache Kafka 4.0, you suggested it would include: > > > > > > > > > > - log4j2 migration > > > > > - idempotency enablement cleanups > > > > > - removal of Java 8 and Scala 2.12 support > > > > > - removal of MirrorMaker1 > > > > > > > > > > It's too soon to remove Java 8/Scala 2.12 support, so I don't think > > > that > > > > > would work. The other things hardly justify a major release so > soon. > > > Have > > > > > we considered adjusting the existing log4j 2 PR so that both > libraries > > > > > versions are supported for a period of time? Since reload4j doesn't > > > have > > > > > the CVEs, this would be acceptable and would avoid a premature 4.0 > > > > release. > > > > > I expect 4.0 to be the release after 3.4 or 3.5 given where we are > > > right > > > > > now. > > > > > > > > > > Ismael > > > > > > > > > > On Wed, Mar 23, 2022 at 4:43 AM Mickael Maison < > > > mickael.mai...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hi Ismael, > > > > > > > > > > > > About 2) > > > > > > We can't keep shipping new releases with dependencies that have > CVEs. > > > > > > This is negatively impacting the project and eroding the hard > earned > > > > > > trust we have from our users. Kafka is known to be a robust, > reliable > > > > > > and up to date project. > > > > > > > > > > > > With that in mind, and since clearly at this point we're not > going to > > > > > > update to log4j2 in 3.2.0, I too would be in favor of tactically > > > > > > adopting reload4j in 3.2.0. This would allow 3.2.0 to release > without > > > > > > any known CVEs and surely make the life of many users better! > > > > > > > > > > > > Now regarding log4j2. I still consider there's value in adopting > > > > > > log4j2 (Apache project, plugin ecosystem, reconfiguration > support) > > > and > > > > > > I'd like to see it happen as soon as possible. If unfortunately > there > > > > > > are compatibility issues, I agree that we can't force breakage > in a > > > > > > minor release. We've always put a lot of attention into > preserving > > > > > > compatibility, we should not suddenly stop doing it. So it makes > > > sense > > > > > > to wait for the next major release. > > > > > > > > > > > > Currently in many minds, 4.0 is kind of associated with the > removal > > > of > > > > > > ZooKeeper. At this stage, it's still unclear when this will be > ready > > > > > > and even if I'm optimistic it's still at the very least 6 to 9 > months > > > > > > away. The code changes to migrate to log4j2 are not trivial and > > > > > > there's certainly a high cost in maintaining then outside of > trunk > > > for > > > > > > many months. Dongjin has done a stellar work so far in regularly > > > > > > updating his PRs since this KIP was started back in 2020, but we > > > can't > > > > > > ask him to just keep doing it for another unknown amount of time. > > > > > > > > > > > > What about if the next release is 4.0? Even if it's light on > > > features, > > > > > > it would enable us to do quite a few cleanups and migrate to > log4j2. > > > > > > Then the removal of ZooKeeper can happen in a future major > release > > > > > > when it's ready. > > > > > > > > > > > > 4.0 would include: > > > > > > - log4j2 migration > > > > > > - idempotency enablement cleanups > > > > > > - removal of Java 8 and Scala 2.12 support > > > > > > - removal of MirrorMaker1 > > > > > > > > > > > > So I propose to adopt reload4j in Kafka 3.2 and make the next > release > > > > > > 4.0. Let me know what you think. > > > > > > > > > > > > Thanks, > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 21, 2022 at 4:33 PM Ismael Juma <ism...@juma.me.uk> > > > wrote: > > > > > > > > > > > > > > Hi Edoardo, > > > > > > > > > > > > > > Thanks for the information. That's definitely useful. A couple > of > > > > > > questions > > > > > > > for you and the rest of the group: > > > > > > > > > > > > > > 1. Did you test the branch using log4j 1.x configs? > > > > > > > 2. Given the release of https://github.com/qos-ch/reload4j, > does > > > it > > > > > > really > > > > > > > make sense to force breakage on users in a minor release? > Would it > > > > not > > > > > be > > > > > > > better to use reload4j in Kafka 3.2 and log4j 2 in Kafka 4.0? > > > > > > > > > > > > > > Thanks, > > > > > > > Ismael > > > > > > > > > > > > > > On Mon, Mar 21, 2022 at 8:16 AM Edoardo Comar < > eco...@uk.ibm.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi Ismael and Luke, > > > > > > > > we've tested Dongjin code - porting her preview releases and > PR > > > to > > > > > > > > different Kafka code levels (2.8.1+, 3.1.0+, trunk). > > > > > > > > We're happy with it and would love it if her PR was merged in > > > > 3.2.0. > > > > > > > > > > > > > > > > To chime in on the issue of compatibility, as we have > experienced > > > > it, > > > > > > the > > > > > > > > main limitation of the log4j-1.2-api.jar 'bridge' jar is in > the > > > > > > support for > > > > > > > > custom Appenders, Filters and Layouts. > > > > > > > > If you're using such components, they may need to be > rewritten to > > > > the > > > > > > > > Log4j2 spec and correspondingly use the configuration file in > > > > log4j2 > > > > > > format > > > > > > > > (and referenced with the log4j2 system property). > > > > > > > > Details at > > > > > > > > > > > > > > > > > > > > > > > > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#ConfigurationCompatibility > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#Log4j1.2BridgeLimitations > > > > > > > > > > > > > > > > I think that the above information should find its way in the > > > KIP's > > > > > > > > compatibility section. > > > > > > > > > > > > > > > > HTH > > > > > > > > Edo > > > > > > > > -------------------------------------------------- > > > > > > > > Edoardo Comar > > > > > > > > Event Streams for IBM Cloud > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > From: Luke Chen <show...@gmail.com> > > > > > > > > Sent: 18 March 2022 07:57 > > > > > > > > To: dev <dev@kafka.apache.org> > > > > > > > > Subject: [EXTERNAL] Re: [VOTE] KIP-653: Upgrade log4j to > log4j2 > > > > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > I know there are some discussions about the compatibility > issue. > > > > > > > > Could you help answer this question? > > > > > > > > > > > > > > > > Thank you. > > > > > > > > Luke > > > > > > > > > > > > > > > > On Fri, Mar 18, 2022 at 3:32 AM Ismael Juma < > ism...@juma.me.uk> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > The KIP compatibility section does not include enough > detail. I > > > > am > > > > > > > > puzzled > > > > > > > > > how we voted +1 given that. I noticed that Colin indicated > it > > > > would > > > > > > only > > > > > > > > be > > > > > > > > > acceptable in a major release unless the new version was > fully > > > > > > compatible > > > > > > > > > (which it is not). Can we clarify what we actually voted > for > > > > here? > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020 at 6:41 PM Dongjin Lee < > > > dong...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > - Binding: +3 (Gwen, John, Colin) > > > > > > > > > > - Non-binding: +1 (David, Tom) > > > > > > > > > > > > > > > > > > > > This KIP is now accepted. Thanks for your votes! > > > > > > > > > > > > > > > > > > > > @Colin Sure, I have some plan for providing a > compatibility > > > > > > preview. > > > > > > > > > Let's > > > > > > > > > > continue in the discussion thread. > > > > > > > > > > > > > > > > > > > > All other voters not in KIP-676 Vote thread: KIP-676 (by > Tom) > > > > is > > > > > a > > > > > > > > > > prerequisite of this KIP. Please have a look at that > proposal > > > > and > > > > > > vote > > > > > > > > > for > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020 at 9:17 PM Colin McCabe < > > > > cmcc...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 (binding). I think we should consider doing this > in 3.0 > > > > > > rather > > > > > > > > than > > > > > > > > > > > 2.8, though, unless we are really confident that it is > 100% > > > > > > > > compatible. > > > > > > > > > > > > > > > > > > > > > > I wasn't able to find much information on how > compatible > > > the > > > > > new > > > > > > API > > > > > > > > > > > bridge is, but the log4j website does have this: > > > > > > > > > > > > > > > > > > > > > > > Basic compatibility with Log4j 1.x is provided > through > > > the > > > > > > > > > log4j12-api > > > > > > > > > > > component, > > > > > > > > > > > > however it does not implement some of the very > > > > implementation > > > > > > > > > specific > > > > > > > > > > > > classes and methods > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 9, 2020, at 02:51, Tom Bentley wrote: > > > > > > > > > > > > +1 non-binding. > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your efforts on this Dongjin. > > > > > > > > > > > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 6:45 AM Dongjin Lee < > > > > > dong...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > > > > - Non-binding: +1 (David) > > > > > > > > > > > > > > > > > > > > > > > > > > Now we need one more binding +1. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 1:37 AM David Jacot < > > > > > > > > david.ja...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this, Dongjin! > > > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP looks good to me. I’m +1 (non-binding). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > David > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le mar. 6 oct. 2020 à 17:23, Dongjin Lee < > > > > > > dong...@apache.org> > > > > > > > > a > > > > > > > > > > > écrit : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > > > > > > - Non-binding: 0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Oct 3, 2020 at 10:51 AM John Roesler < > > > > > > > > > > vvcep...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP, Dongjin! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’ve just reviewed the KIP document, and it > looks > > > > > good > > > > > > to > > > > > > > > me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2020, at 19:11, Gwen Shapira > > > wrote: > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A very welcome update :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2020 at 9:09 AM Dongjin > Lee < > > > > > > > > > > > dong...@apache.org> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here I open the vote for KIP-653: Upgrade > > > log4j > > > > > to > > > > > > > > > log4j2. > > > > > > > > > > It > > > > > > > > > > > > > > > replaces > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > obsolete log4j logging library into the > > > current > > > > > > > > standard, > > > > > > > > > > > log4j2, > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > maintaining backward-compatibility. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > > > > <https://github.com/dongjinleekr > >keybase: > > > > > > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > > > > > > <https://keybase.io/dongjinleekr > >linkedin: > > > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > > > > > > >speakerdeck: > > > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > Gwen Shapira > > > > > > > > > > > > > > > > > Engineering Manager | Confluent > > > > > > > > > > > > > > > > > 650.450.2760 | @gwenshap > > > > > > > > > > > > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > > > > >speakerdeck: > > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > github.com/dongjinleekr > > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > >speakerdeck: > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > github.com/dongjinleekr > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > <https://github.com/dongjinleekr>keybase: > > > https://keybase.io/dongjinleekr > > > > <https://keybase.io/dongjinleekr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > > speakerdeck.com/dongjin > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > -- > > *Dongjin Lee* > > > > *A hitchhiker in the mathematical world.* > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > <https://github.com/dongjinleekr>keybase: > https://keybase.io/dongjinleekr > > <https://keybase.io/dongjinleekr>linkedin: > kr.linkedin.com/in/dongjinleekr > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > > <https://speakerdeck.com/dongjin>* >
