Hi Colin, Regarding KIP-719, I think need it to land in 3.8 if we want to remove the appender in 4.0. I also just noticed the log4j's KafkaAppender is being deprecated in log4j2 and will not be part of log4j3.
For KIP-653, as I said, my point was to gauge interest in getting it done. While it may not be a "must-do" to keep Kafka working, we can only do this type of change in major releases. So if we don't do it now, it won't happen for a few more years. Regarding log4j3, even though the website states it requires Java 11 [1], it seems the latest beta release requires Java 17 so it's not something we'll be able to adopt now. 0: https://github.com/apache/logging-log4j2/issues/1951 1: https://logging.apache.org/log4j/3.x/#requirements Thanks, Mickael On Fri, Jan 5, 2024 at 12:18 AM Colin McCabe <cmcc...@apache.org> wrote: > > Hi Mickael, > > Thanks for bringing this up. > > The main motivation given in KIP-653 for moving to log4j 2.x is that log4j > 1.x is no longer supported. But since we moved to reload4j, which is still > supported, that isn't a concern any longer. > > To be clear, I'm not saying we shouldn't upgrade, but I'm just trying to > explain why I think there hasn't been as much interest in this lately. I see > this as a "cool feature" rather than as a must-do. > > If we still want to do this for 4.0, it would be good to understand whether > there's any work that has to land in 3.8. Do we have to get KIP-719 into 3.8 > so that we have a reasonable deprecation period? > > Also, if we do upgrade, I agree with Ismael that we should consider going to > log4j3. Assuming they have a non-beta release by the time 4.0 is ready. > > best, > Colin > > On Thu, Jan 4, 2024, at 03:08, Mickael Maison wrote: > > Hi Ismael, > > > > Yes both KIPs have been voted. > > My point, which admittedly wasn't clear, was to gauge the interest in > > getting them done and if so identifying people to drive these tasks. > > > > KIP-719 shouldn't require too much more work to complete. There's a PR > > [0] which is relatively straightforward. I pinged Lee Dongjin. > > KIP-653 is more involved and depends on KIP-719. There's also a PR [1] > > which is pretty large. > > > > Yes log4j3 was on my mind as it's expected to be compatible with > > log4j2 and bring significant improvements. > > > > 0: https://github.com/apache/kafka/pull/10244 > > 1: https://github.com/apache/kafka/pull/7898 > > > > Thanks, > > Mickael > > > > On Thu, Jan 4, 2024 at 11:34 AM Ismael Juma <m...@ismaeljuma.com> wrote: > >> > >> Hi Mickael, > >> > >> Given that KIP-653 was accepted, the current position is that we would move > >> to log4j2 - provided that someone is available to drive that. It's also > >> worth noting that log4j3 is now a thing (but not yet final): > >> > >> https://logging.apache.org/log4j/3.x/ > >> > >> Ismael > >> > >> On Thu, Jan 4, 2024 at 2:15 AM Mickael Maison <mickael.mai...@gmail.com> > >> wrote: > >> > >> > Hi, > >> > > >> > I've not seen replies about log4j2. > >> > The plan was to deprecated the appender (KIP-719) and switch to log4j2 > >> > (KIP-653). > >> > > >> > While reload4j works well, I'd still be in favor of switching to > >> > log4j2 in Kafka 4.0. > >> > > >> > Thanks, > >> > Mickael > >> > > >> > On Fri, Dec 29, 2023 at 2:19 AM Colin McCabe <co...@cmccabe.xyz> wrote: > >> > > > >> > > Hi all, > >> > > > >> > > Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for > >> > a Kafka 3.8.x release" email thread. > >> > > > >> > > Colin > >> > > > >> > > > >> > > On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote: > >> > > > Hi Divij, > >> > > > > >> > > > Thanks for the feedback. I agree that having a 3.8 release is > >> > > > beneficial but some of the comments in this message are inaccurate > >> > > > and > >> > > > could mislead the community and users. > >> > > > > >> > > > On Thu, Dec 21, 2023 at 7:00 AM Divij Vaidya > >> > > > <divijvaidy...@gmail.com> > >> > wrote: > >> > > >> 1\ Durability/availability bugs in kraft - Even though kraft has > >> > > >> been > >> > > >> around for a while, we keep finding bugs that impact availability > >> > > >> and > >> > data > >> > > >> durability in it almost with every release [1] [2]. It's a complex > >> > feature > >> > > >> and such bugs are expected during the stabilization phase. But we > >> > can't > >> > > >> remove the alternative until we see stabilization in kraft i.e. no > >> > > >> new > >> > > >> stability/durability bugs for at least 2 releases. > >> > > > > >> > > > I took a look at both of these issues and neither of them are bugs > >> > > > that affect KRaft's durability and availability. > >> > > > > >> > > >> [1] https://issues.apache.org/jira/browse/KAFKA-15495 > >> > > > > >> > > > This issue is not specific to KRaft and has been an issue in Apache > >> > > > Kafka since the ISR leader election and replication algorithm was > >> > > > added to Apache Kafka. I acknowledge that this misunderstanding is > >> > > > partially due to the Jira description which insinuates that this only > >> > > > applies to KRaft which is not true. > >> > > > > >> > > >> [2] https://issues.apache.org/jira/browse/KAFKA-15489 > >> > > > > >> > > > First, technically this issue was not first discovered in some recent > >> > > > release. This issue was identified by me back in January of 2022: > >> > > > https://issues.apache.org/jira/browse/KAFKA-13621. I decided to lower > >> > > > the priority as it requires a very specific network partition where > >> > > > the controllers are partitioned from the current leader but the > >> > > > brokers are not. > >> > > > > >> > > > This is not a durability bug as the KRaft cluster metadata partition > >> > > > leader will not be able to advance the HWM and hence commit records. > >> > > > > >> > > > Regarding availability, The KRaft's cluster metadata partition favors > >> > > > consistency and partition tolerance versus availability from CAP. > >> > > > This > >> > > > is by design and not a bug in the protocol or implementation. > >> > > > > >> > > >> 2\ Parity with Zk - There are also pending bugs [3] which are in the > >> > > >> category of Zk parity. Removing Zk from Kafka without having full > >> > feature > >> > > >> parity with Zk will leave some Kafka users with no upgrade path. > >> > > >> 3\ Test coverage - We also don't have sufficient test coverage for > >> > kraft > >> > > >> since quite a few tests are Zk only at this stage. > >> > > >> > >> > > >> Given these concerns, I believe we need to reach 100% Zk parity and > >> > allow > >> > > >> new feature stabilisation (such as scram, JBOD) for at least 1 > >> > > >> version > >> > > >> (maybe more if we find bugs in that feature) before we remove Zk. I > >> > also > >> > > >> agree with the point of view that we can't delay 4.0 indefinitely > >> > > >> and > >> > we > >> > > >> need a clear cut line. > >> > > > > >> > > > There seems to be some misunderstanding regarding Apache Kafka > >> > > > versioning scheme. Minor versions (e.g. 3.x) are needed for feature > >> > > > releases like new RPCs and configurations. They are not needed for > >> > > > bug > >> > > > fixes. Bug fixes can and should be done in patch releases (e.g. > >> > > > 3.7.x). > >> > > > > >> > > > This means that you don't need a 3.8 or 3.9 release to fix a bug in > >> > Kafka. > >> > > > > >> > > > Thanks! > >> > > > -- > >> > > > -José > >> >
