Hi Dongjin, We really appreciate the super valuable work you've been doing here. Do we have evidence that customers don't use custom filters/layouts?
Ismael On Wed, Mar 23, 2022 at 7:53 AM Dongjin Lee <dong...@apache.org> wrote: > Hi Mikael, Edoardo and Ismael, > > Sorry for being late. Frankly, I thought KIP-653 is not a breaking change > since (as Edoardo stated) unless the user uses custom filters or layouts, > log4j-1.2-api.jar 'bridge' jar can handle the cases. It is why the > 'Compatibility, Deprecation, and Migration Plan' section of the document is > so brief. (As far as I know, it is such a rare case, and I thought it would > not be so problematic.) > > I have no firm position on the release plan of this feature. Regardless of > whether the community decides to put off the adoption of log4j2 to 4.0, I > will maintain the PR and the preview releases up-to-date as far as possible > - although it can be significantly late for my main job. The decision is > totally up to the community - No matter how it is decided, I will follow. > > Best, > Dongjin > > p.s. @Edoardo I'm HIM. (HAHA) > > On Wed, Mar 23, 2022 at 10:12 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Mickael, > > > > Thanks for your feedback. I agree with the importance of fixing the CVEs > > and also of not breaking compatibility in a critical layer. Regarding > > Apache Kafka 4.0, you suggested it would include: > > > > - log4j2 migration > > - idempotency enablement cleanups > > - removal of Java 8 and Scala 2.12 support > > - removal of MirrorMaker1 > > > > It's too soon to remove Java 8/Scala 2.12 support, so I don't think that > > would work. The other things hardly justify a major release so soon. Have > > we considered adjusting the existing log4j 2 PR so that both libraries > > versions are supported for a period of time? Since reload4j doesn't have > > the CVEs, this would be acceptable and would avoid a premature 4.0 > release. > > I expect 4.0 to be the release after 3.4 or 3.5 given where we are right > > now. > > > > Ismael > > > > On Wed, Mar 23, 2022 at 4:43 AM Mickael Maison <mickael.mai...@gmail.com > > > > wrote: > > > > > Hi Ismael, > > > > > > About 2) > > > We can't keep shipping new releases with dependencies that have CVEs. > > > This is negatively impacting the project and eroding the hard earned > > > trust we have from our users. Kafka is known to be a robust, reliable > > > and up to date project. > > > > > > With that in mind, and since clearly at this point we're not going to > > > update to log4j2 in 3.2.0, I too would be in favor of tactically > > > adopting reload4j in 3.2.0. This would allow 3.2.0 to release without > > > any known CVEs and surely make the life of many users better! > > > > > > Now regarding log4j2. I still consider there's value in adopting > > > log4j2 (Apache project, plugin ecosystem, reconfiguration support) and > > > I'd like to see it happen as soon as possible. If unfortunately there > > > are compatibility issues, I agree that we can't force breakage in a > > > minor release. We've always put a lot of attention into preserving > > > compatibility, we should not suddenly stop doing it. So it makes sense > > > to wait for the next major release. > > > > > > Currently in many minds, 4.0 is kind of associated with the removal of > > > ZooKeeper. At this stage, it's still unclear when this will be ready > > > and even if I'm optimistic it's still at the very least 6 to 9 months > > > away. The code changes to migrate to log4j2 are not trivial and > > > there's certainly a high cost in maintaining then outside of trunk for > > > many months. Dongjin has done a stellar work so far in regularly > > > updating his PRs since this KIP was started back in 2020, but we can't > > > ask him to just keep doing it for another unknown amount of time. > > > > > > What about if the next release is 4.0? Even if it's light on features, > > > it would enable us to do quite a few cleanups and migrate to log4j2. > > > Then the removal of ZooKeeper can happen in a future major release > > > when it's ready. > > > > > > 4.0 would include: > > > - log4j2 migration > > > - idempotency enablement cleanups > > > - removal of Java 8 and Scala 2.12 support > > > - removal of MirrorMaker1 > > > > > > So I propose to adopt reload4j in Kafka 3.2 and make the next release > > > 4.0. Let me know what you think. > > > > > > Thanks, > > > Mickael > > > > > > > > > > > > On Mon, Mar 21, 2022 at 4:33 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > > Hi Edoardo, > > > > > > > > Thanks for the information. That's definitely useful. A couple of > > > questions > > > > for you and the rest of the group: > > > > > > > > 1. Did you test the branch using log4j 1.x configs? > > > > 2. Given the release of https://github.com/qos-ch/reload4j, does it > > > really > > > > make sense to force breakage on users in a minor release? Would it > not > > be > > > > better to use reload4j in Kafka 3.2 and log4j 2 in Kafka 4.0? > > > > > > > > Thanks, > > > > Ismael > > > > > > > > On Mon, Mar 21, 2022 at 8:16 AM Edoardo Comar <eco...@uk.ibm.com> > > wrote: > > > > > > > > > Hi Ismael and Luke, > > > > > we've tested Dongjin code - porting her preview releases and PR to > > > > > different Kafka code levels (2.8.1+, 3.1.0+, trunk). > > > > > We're happy with it and would love it if her PR was merged in > 3.2.0. > > > > > > > > > > To chime in on the issue of compatibility, as we have experienced > it, > > > the > > > > > main limitation of the log4j-1.2-api.jar 'bridge' jar is in the > > > support for > > > > > custom Appenders, Filters and Layouts. > > > > > If you're using such components, they may need to be rewritten to > the > > > > > Log4j2 spec and correspondingly use the configuration file in > log4j2 > > > format > > > > > (and referenced with the log4j2 system property). > > > > > Details at > > > > > > > > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#ConfigurationCompatibility > > > > > and > > > > > > > > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#Log4j1.2BridgeLimitations > > > > > > > > > > I think that the above information should find its way in the KIP's > > > > > compatibility section. > > > > > > > > > > HTH > > > > > Edo > > > > > -------------------------------------------------- > > > > > Edoardo Comar > > > > > Event Streams for IBM Cloud > > > > > > > > > > > > > > > ________________________________ > > > > > From: Luke Chen <show...@gmail.com> > > > > > Sent: 18 March 2022 07:57 > > > > > To: dev <dev@kafka.apache.org> > > > > > Subject: [EXTERNAL] Re: [VOTE] KIP-653: Upgrade log4j to log4j2 > > > > > > > > > > Hi Dongjin, > > > > > > > > > > I know there are some discussions about the compatibility issue. > > > > > Could you help answer this question? > > > > > > > > > > Thank you. > > > > > Luke > > > > > > > > > > On Fri, Mar 18, 2022 at 3:32 AM Ismael Juma <ism...@juma.me.uk> > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > The KIP compatibility section does not include enough detail. I > am > > > > > puzzled > > > > > > how we voted +1 given that. I noticed that Colin indicated it > would > > > only > > > > > be > > > > > > acceptable in a major release unless the new version was fully > > > compatible > > > > > > (which it is not). Can we clarify what we actually voted for > here? > > > > > > > > > > > > Ismael > > > > > > > > > > > > On Wed, Oct 21, 2020 at 6:41 PM Dongjin Lee <dong...@apache.org> > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > - Binding: +3 (Gwen, John, Colin) > > > > > > > - Non-binding: +1 (David, Tom) > > > > > > > > > > > > > > This KIP is now accepted. Thanks for your votes! > > > > > > > > > > > > > > @Colin Sure, I have some plan for providing a compatibility > > > preview. > > > > > > Let's > > > > > > > continue in the discussion thread. > > > > > > > > > > > > > > All other voters not in KIP-676 Vote thread: KIP-676 (by Tom) > is > > a > > > > > > > prerequisite of this KIP. Please have a look at that proposal > and > > > vote > > > > > > for > > > > > > > it. > > > > > > > > > > > > > > Best, > > > > > > > Dongjin > > > > > > > > > > > > > > On Wed, Oct 21, 2020 at 9:17 PM Colin McCabe < > cmcc...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 (binding). I think we should consider doing this in 3.0 > > > rather > > > > > than > > > > > > > > 2.8, though, unless we are really confident that it is 100% > > > > > compatible. > > > > > > > > > > > > > > > > I wasn't able to find much information on how compatible the > > new > > > API > > > > > > > > bridge is, but the log4j website does have this: > > > > > > > > > > > > > > > > > Basic compatibility with Log4j 1.x is provided through the > > > > > > log4j12-api > > > > > > > > component, > > > > > > > > > however it does not implement some of the very > implementation > > > > > > specific > > > > > > > > > classes and methods > > > > > > > > > > > > > > > > best, > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 9, 2020, at 02:51, Tom Bentley wrote: > > > > > > > > > +1 non-binding. > > > > > > > > > > > > > > > > > > Thanks for your efforts on this Dongjin. > > > > > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 6:45 AM Dongjin Lee < > > dong...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > - Non-binding: +1 (David) > > > > > > > > > > > > > > > > > > > > Now we need one more binding +1. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 1:37 AM David Jacot < > > > > > david.ja...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for driving this, Dongjin! > > > > > > > > > > > > > > > > > > > > > > The KIP looks good to me. I’m +1 (non-binding). > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > David > > > > > > > > > > > > > > > > > > > > > > Le mar. 6 oct. 2020 à 17:23, Dongjin Lee < > > > dong...@apache.org> > > > > > a > > > > > > > > écrit : > > > > > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > > > - Non-binding: 0 > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Oct 3, 2020 at 10:51 AM John Roesler < > > > > > > > vvcep...@apache.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP, Dongjin! > > > > > > > > > > > > > > > > > > > > > > > > > > I’ve just reviewed the KIP document, and it looks > > good > > > to > > > > > me. > > > > > > > > > > > > > > > > > > > > > > > > > > I’m +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > John > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2020, at 19:11, Gwen Shapira wrote: > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > A very welcome update :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2020 at 9:09 AM Dongjin Lee < > > > > > > > > dong...@apache.org> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here I open the vote for KIP-653: Upgrade log4j > > to > > > > > > log4j2. > > > > > > > It > > > > > > > > > > > > replaces > > > > > > > > > > > > > the > > > > > > > > > > > > > > > obsolete log4j logging library into the current > > > > > standard, > > > > > > > > log4j2, > > > > > > > > > > > > with > > > > > > > > > > > > > > > maintaining backward-compatibility. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > > > >speakerdeck: > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Gwen Shapira > > > > > > > > > > > > > > Engineering Manager | Confluent > > > > > > > > > > > > > > 650.450.2760 | @gwenshap > > > > > > > > > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > github.com/dongjinleekr > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > >speakerdeck: > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > github.com/dongjinleekr > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > https://keybase.io/dongjinleekr > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > speakerdeck.com/dongjin > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* >
