Hi Ismael, About 2) We can't keep shipping new releases with dependencies that have CVEs. This is negatively impacting the project and eroding the hard earned trust we have from our users. Kafka is known to be a robust, reliable and up to date project.
With that in mind, and since clearly at this point we're not going to update to log4j2 in 3.2.0, I too would be in favor of tactically adopting reload4j in 3.2.0. This would allow 3.2.0 to release without any known CVEs and surely make the life of many users better! Now regarding log4j2. I still consider there's value in adopting log4j2 (Apache project, plugin ecosystem, reconfiguration support) and I'd like to see it happen as soon as possible. If unfortunately there are compatibility issues, I agree that we can't force breakage in a minor release. We've always put a lot of attention into preserving compatibility, we should not suddenly stop doing it. So it makes sense to wait for the next major release. Currently in many minds, 4.0 is kind of associated with the removal of ZooKeeper. At this stage, it's still unclear when this will be ready and even if I'm optimistic it's still at the very least 6 to 9 months away. The code changes to migrate to log4j2 are not trivial and there's certainly a high cost in maintaining then outside of trunk for many months. Dongjin has done a stellar work so far in regularly updating his PRs since this KIP was started back in 2020, but we can't ask him to just keep doing it for another unknown amount of time. What about if the next release is 4.0? Even if it's light on features, it would enable us to do quite a few cleanups and migrate to log4j2. Then the removal of ZooKeeper can happen in a future major release when it's ready. 4.0 would include: - log4j2 migration - idempotency enablement cleanups - removal of Java 8 and Scala 2.12 support - removal of MirrorMaker1 So I propose to adopt reload4j in Kafka 3.2 and make the next release 4.0. Let me know what you think. Thanks, Mickael On Mon, Mar 21, 2022 at 4:33 PM Ismael Juma <ism...@juma.me.uk> wrote: > > Hi Edoardo, > > Thanks for the information. That's definitely useful. A couple of questions > for you and the rest of the group: > > 1. Did you test the branch using log4j 1.x configs? > 2. Given the release of https://github.com/qos-ch/reload4j, does it really > make sense to force breakage on users in a minor release? Would it not be > better to use reload4j in Kafka 3.2 and log4j 2 in Kafka 4.0? > > Thanks, > Ismael > > On Mon, Mar 21, 2022 at 8:16 AM Edoardo Comar <eco...@uk.ibm.com> wrote: > > > Hi Ismael and Luke, > > we've tested Dongjin code - porting her preview releases and PR to > > different Kafka code levels (2.8.1+, 3.1.0+, trunk). > > We're happy with it and would love it if her PR was merged in 3.2.0. > > > > To chime in on the issue of compatibility, as we have experienced it, the > > main limitation of the log4j-1.2-api.jar 'bridge' jar is in the support for > > custom Appenders, Filters and Layouts. > > If you're using such components, they may need to be rewritten to the > > Log4j2 spec and correspondingly use the configuration file in log4j2 format > > (and referenced with the log4j2 system property). > > Details at > > https://logging.apache.org/log4j/2.x/manual/migration.html#ConfigurationCompatibility > > and > > https://logging.apache.org/log4j/2.x/manual/migration.html#Log4j1.2BridgeLimitations > > > > I think that the above information should find its way in the KIP's > > compatibility section. > > > > HTH > > Edo > > -------------------------------------------------- > > Edoardo Comar > > Event Streams for IBM Cloud > > > > > > ________________________________ > > From: Luke Chen <show...@gmail.com> > > Sent: 18 March 2022 07:57 > > To: dev <dev@kafka.apache.org> > > Subject: [EXTERNAL] Re: [VOTE] KIP-653: Upgrade log4j to log4j2 > > > > Hi Dongjin, > > > > I know there are some discussions about the compatibility issue. > > Could you help answer this question? > > > > Thank you. > > Luke > > > > On Fri, Mar 18, 2022 at 3:32 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi all, > > > > > > The KIP compatibility section does not include enough detail. I am > > puzzled > > > how we voted +1 given that. I noticed that Colin indicated it would only > > be > > > acceptable in a major release unless the new version was fully compatible > > > (which it is not). Can we clarify what we actually voted for here? > > > > > > Ismael > > > > > > On Wed, Oct 21, 2020 at 6:41 PM Dongjin Lee <dong...@apache.org> wrote: > > > > > > > Hi All, > > > > > > > > As of present: > > > > > > > > - Binding: +3 (Gwen, John, Colin) > > > > - Non-binding: +1 (David, Tom) > > > > > > > > This KIP is now accepted. Thanks for your votes! > > > > > > > > @Colin Sure, I have some plan for providing a compatibility preview. > > > Let's > > > > continue in the discussion thread. > > > > > > > > All other voters not in KIP-676 Vote thread: KIP-676 (by Tom) is a > > > > prerequisite of this KIP. Please have a look at that proposal and vote > > > for > > > > it. > > > > > > > > Best, > > > > Dongjin > > > > > > > > On Wed, Oct 21, 2020 at 9:17 PM Colin McCabe <cmcc...@apache.org> > > wrote: > > > > > > > > > +1 (binding). I think we should consider doing this in 3.0 rather > > than > > > > > 2.8, though, unless we are really confident that it is 100% > > compatible. > > > > > > > > > > I wasn't able to find much information on how compatible the new API > > > > > bridge is, but the log4j website does have this: > > > > > > > > > > > Basic compatibility with Log4j 1.x is provided through the > > > log4j12-api > > > > > component, > > > > > > however it does not implement some of the very implementation > > > specific > > > > > > classes and methods > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > > > > > > On Fri, Oct 9, 2020, at 02:51, Tom Bentley wrote: > > > > > > +1 non-binding. > > > > > > > > > > > > Thanks for your efforts on this Dongjin. > > > > > > > > > > > > Tom > > > > > > > > > > > > On Wed, Oct 7, 2020 at 6:45 AM Dongjin Lee <dong...@apache.org> > > > wrote: > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > - Non-binding: +1 (David) > > > > > > > > > > > > > > Now we need one more binding +1. > > > > > > > > > > > > > > Thanks, > > > > > > > Dongjin > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 1:37 AM David Jacot < > > david.ja...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Thanks for driving this, Dongjin! > > > > > > > > > > > > > > > > The KIP looks good to me. I’m +1 (non-binding). > > > > > > > > > > > > > > > > Best, > > > > > > > > David > > > > > > > > > > > > > > > > Le mar. 6 oct. 2020 à 17:23, Dongjin Lee <dong...@apache.org> > > a > > > > > écrit : > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > - Non-binding: 0 > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > On Sat, Oct 3, 2020 at 10:51 AM John Roesler < > > > > vvcep...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks for the KIP, Dongjin! > > > > > > > > > > > > > > > > > > > > I’ve just reviewed the KIP document, and it looks good to > > me. > > > > > > > > > > > > > > > > > > > > I’m +1 (binding) > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > John > > > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2020, at 19:11, Gwen Shapira wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > > > A very welcome update :) > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2020 at 9:09 AM Dongjin Lee < > > > > > dong...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > Here I open the vote for KIP-653: Upgrade log4j to > > > log4j2. > > > > It > > > > > > > > > replaces > > > > > > > > > > the > > > > > > > > > > > > obsolete log4j logging library into the current > > standard, > > > > > log4j2, > > > > > > > > > with > > > > > > > > > > > > maintaining backward-compatibility. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > github.com/dongjinleekr > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Gwen Shapira > > > > > > > > > > > Engineering Manager | Confluent > > > > > > > > > > > 650.450.2760 | @gwenshap > > > > > > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > https://keybase.io/dongjinleekr > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > speakerdeck.com/dongjin > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > <https://github.com/dongjinleekr >keybase: > > > https://keybase.io/dongjinleekr > > > > <https://keybase.io/dongjinleekr >linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > speakerdeck.com/dongjin > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > >
