Hi folks, >a) I think this makes sense. But I think this shouldn't block this KIP, it can be done after moving to log4j2.
Absolutely. >However, this limitation may change in the future - slf4j will support dynamic logging levels in 2.0: This would be awesome, let's keep an eye on this. >Kafka avoided all the log4shell vulnerabilities exactly because we DIDN'T move to log4j 2.x. My first reaction was how lucky we are (and all Kafka users) that we didn't have to do anything on that weekend! Blessed are the lazy. :) Viktor On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote: > > Hi Dongjin, > > > > We're also looking into this matter as our stack was also affected by all > > the log4j hell and users increasingly pushing us to upgrade to log4j2 or > > logback because of the existing vulnerabilities in log4j1. > > Regarding the points raised by Haruki: > > > > I've heard the same thing from other people -- that there is now more > interest in moving from log4j1.x to log4j2.x, after all the recent > vulnerabilities in the latter. I found this a bit strange. Kafka avoided > all the log4shell vulnerabilities exactly because we DIDN'T move to log4j > 2.x. (Yes, I am aware that there is a longstanding vulnerability in that > one log sink in log4j 1.x, but you can just not use that one.) > > I haven't thought about this very hard. Maybe it's still a good idea to > move to log4j2. But it's odd that nobody is commenting about how in this > case, not updating actually prevented a major security incident for Kafka. > > best, > Colin > > > > > a) In my opinion the best would be to make the dynamic logger support > > (Log4jController and LoggingResource) pluggable for log4j2 and logback > (so > > an interface could be used to define the dynamic logging control methods > > and a config to specify the implementation). That way we're not bound to > > either logback or log4j and seems like a low-effort thing to do. > > Additionally this could be used in Connect too in LoggingResource. > > > > b) I think testing dependencies aren't that important from the user > > perspective, it's fine to either use log4j2 or logback, whichever is > > easier. Kafka is either used from the distribution (tgz) or pulled in > > through maven, but test dependencies shouldn't be exposed to the world. > > > > c) I would support deprecating the appender in favor of the log4j2 Kafka > > appender. VerifiableLog4jAppender is intended as a testing tool anyway, > so > > I think it's less important to change this to logback. > > > > Future vulnerabilities will always be found in either logback or log4j2 > or > > any other logging framework, so I think the safest approach is to allow > > users to choose their implementation, while in tests I think we're free > to > > use whatever we want as that shouldn't be constrained by vulnerabilities. > > > > Viktor > > > > On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada <ocadar...@gmail.com> > wrote: > > > >> Thanks for the clarification. > >> > >> About 2, I wan't aware of those concerns. > >> Let me check them first. > >> > >> > >> Thanks, > >> > >> 2021年12月23日(木) 13:37 Dongjin Lee <dong...@apache.org>: > >> > >> > Hi Haruki, > >> > > >> > > >> > Thanks for organizing the issue. > >> > > >> > > >> > If the community prefers logback, I will gladly change the dependency > and > >> > update the PR. However, it has the following issues: > >> > > >> > > >> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719 > >> are > >> > not released yet. So, using log4j2 (whose recent update pace is so > high) > >> > will not affect the users. > >> > > >> > > >> > 2. To switch to logback, the following features should be reworked: > >> > > >> > > >> > a. Dynamic logger level configuration (core, connect) > >> > > >> > b. Logging tests (streams) > >> > > >> > c. Kafka Appender (tools) > >> > > >> > > >> > a and b are the most challenging ones since there is little > documentation > >> > on how to do this, so it requires analyzing the implementation itself. > >> > (what I actually did with log4j2) About c, logback does not provide a > >> Kafka > >> > Appender so we have to provide an equivalent. > >> > > >> > > >> > It is why I prefer to use log4j2. How do you think? > >> > > >> > > >> > Thanks, > >> > > >> > Dongjin > >> > > >> > > >> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada <ocadar...@gmail.com> > >> wrote: > >> > > >> > > Hi, Dongjin, > >> > > > >> > > Sorry for interrupting the discussion. > >> > > And thank you for your hard work about KIP-653, KIP-719. > >> > > > >> > > I understand that KIP-653 is already accepted so log4j2 is the > choice > >> of > >> > > the Kafka community though, I'm now feeling that logback is a better > >> > choice > >> > > here. > >> > > > >> > > Reasons: > >> > > > >> > > - even after "log4shell", several vulnerabilities found on log4j2 so > >> new > >> > > versions are released and users have to update in high-pace > >> > > * actually, a CVE was also reported for logback (CVE-2021-42550) > >> but > >> > it > >> > > requires edit-permission of the config file for an attacker so it's > >> much > >> > > less threatening > >> > > - log4j1.x and logback are made by same developer (ceki), so > >> > substantially > >> > > the successor of log4j1 is logback rather than log4j2 > >> > > - in Hadoop project, seems similar suggestion was made from a PMC > >> > > * https://issues.apache.org/jira/browse/HADOOP-12956 > >> > > > >> > > > >> > > What do you think about adopting logback instead? > >> > > > >> > > > >> > > Thanks, > >> > > > >> > > 2021年12月21日(火) 18:02 Dongjin Lee <dong...@apache.org>: > >> > > > >> > > > Hi Mickael, > >> > > > > >> > > > > In the meantime, you may want to bump the VOTE thread too. > >> > > > > >> > > > Sure, I just reset the voting thread with a brief context. > >> > > > > >> > > > Thanks, > >> > > > Dongjin > >> > > > > >> > > > On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison < > >> > mickael.mai...@gmail.com > >> > > > > >> > > > wrote: > >> > > > > >> > > > > Thanks Dongjin! > >> > > > > > >> > > > > I'll take a look soon. > >> > > > > In the meantime, you may want to bump the VOTE thread too. > >> > > > > > >> > > > > Best, > >> > > > > Mickael > >> > > > > > >> > > > > > >> > > > > On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee < > dong...@apache.org> > >> > > wrote: > >> > > > > > > >> > > > > > Hi Mickael, > >> > > > > > > >> > > > > > Finally, I did it! As you can see at the PR > >> > > > > > <https://github.com/apache/kafka/pull/10244>, KIP-719 now > uses > >> > > > log4j2's > >> > > > > > Kafka appender, and log4j-appender is not used by the other > >> modules > >> > > > > > anymore. You can see how it will work with KIP-653 at this > >> preview > >> > > > > > < > >> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/ > >> > >, > >> > > > > based > >> > > > > > on Apache Kafka 3.0.0. The proposal document > >> > > > > > < > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender > >> > > > > > > >> > > > > > is also updated accordingly, with its title. > >> > > > > > > >> > > > > > There is a minor issue on log4j2 > >> > > > > > <https://issues.apache.org/jira/browse/LOG4J2-3256>, but it > >> seems > >> > > like > >> > > > > it > >> > > > > > will be resolved soon. > >> > > > > > > >> > > > > > Best, > >> > > > > > Dongjin > >> > > > > > > >> > > > > > On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee < > dong...@apache.org> > >> > > > wrote: > >> > > > > > > >> > > > > > > Hi Mickael, > >> > > > > > > > >> > > > > > > > Can we do step 3 without breaking any compatibility? If so > >> then > >> > > > that > >> > > > > > > sounds like a good idea. > >> > > > > > > > >> > > > > > > As far as I know, the answer is yes; I am now updating my > PR, > >> so > >> > I > >> > > > will > >> > > > > > > notify you as soon as I complete the work. > >> > > > > > > > >> > > > > > > Best, > >> > > > > > > Dongjin > >> > > > > > > > >> > > > > > > On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison < > >> > > > > mickael.mai...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > >> Hi Dongjin, > >> > > > > > >> > >> > > > > > >> Sorry for the late reply. Can we do step 3 without breaking > >> any > >> > > > > > >> compatibility? If so then that sounds like a good idea. > >> > > > > > >> > >> > > > > > >> Thanks, > >> > > > > > >> Mickael > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee < > >> dong...@apache.org > >> > > > >> > > > > wrote: > >> > > > > > >> > > >> > > > > > >> > Hi Mickael, > >> > > > > > >> > > >> > > > > > >> > I also thought over the issue thoroughly and would like > to > >> > > > propose a > >> > > > > > >> minor > >> > > > > > >> > change to your proposal: > >> > > > > > >> > > >> > > > > > >> > 1. Deprecate log4j-appender now > >> > > > > > >> > 2. Document how to migrate into logging-log4j2 > >> > > > > > >> > 3. (Changed) Replace the log4j-appender (in turn log4j > 1.x) > >> > > > > > >> dependencies in > >> > > > > > >> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, > >> > removing > >> > > > > log4j > >> > > > > > >> 1.x > >> > > > > > >> > dependencies. > >> > > > > > >> > 4. (Changed) Remove log4j-appender in Kafka 4.0 > >> > > > > > >> > > >> > > > > > >> > What we need to do for the log4j2 upgrade is just > removing > >> the > >> > > > log4j > >> > > > > > >> > dependencies only, for they can cause a classpath error. > And > >> > > > > actually, > >> > > > > > >> we > >> > > > > > >> > can do it without discontinuing publishing the > >> log4j-appender > >> > > > > artifact. > >> > > > > > >> So, > >> > > > > > >> > I suggest separating the upgrade to log4j2 and removing > the > >> > > > > > >> log4j-appender > >> > > > > > >> > module. > >> > > > > > >> > > >> > > > > > >> > How do you think? If you agree, I will update the KIP and > >> the > >> > PR > >> > > > > > >> > accordingly ASAP. > >> > > > > > >> > > >> > > > > > >> > Thanks, > >> > > > > > >> > Dongjin > >> > > > > > >> > > >> > > > > > >> > On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison < > >> > > > > > >> mickael.mai...@gmail.com> > >> > > > > > >> > wrote: > >> > > > > > >> > > >> > > > > > >> > > Hi Dongjin, > >> > > > > > >> > > > >> > > > > > >> > > Thanks for the clarifications. > >> > > > > > >> > > > >> > > > > > >> > > I wonder if a simpler course of action could be: > >> > > > > > >> > > - Deprecate log4j-appender now > >> > > > > > >> > > - Document how to use logging-log4j2 > >> > > > > > >> > > - Remove log4j-appender and all the log4j dependencies > in > >> > > Kafka > >> > > > > 4.0 > >> > > > > > >> > > > >> > > > > > >> > > This delays KIP-653 till Kafka 4.0 but (so far) Kafka > is > >> not > >> > > > > directly > >> > > > > > >> > > affected by the log4j CVEs. At least this gives us a > clear > >> > and > >> > > > > simple > >> > > > > > >> > > roadmap to follow. > >> > > > > > >> > > > >> > > > > > >> > > What do you think? > >> > > > > > >> > > > >> > > > > > >> > > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee < > >> > > dong...@apache.org > >> > > > > > >> > > > > > >> wrote: > >> > > > > > >> > > > > >> > > > > > >> > > > Hi Mickael, > >> > > > > > >> > > > > >> > > > > > >> > > > I greatly appreciate you for reading the proposal so > >> > > > carefully! > >> > > > > I > >> > > > > > >> wrote > >> > > > > > >> > > it > >> > > > > > >> > > > quite a while ago and rechecked it today. > >> > > > > > >> > > > > >> > > > > > >> > > > > Is the KIP proposing to replace the existing > >> > log4-appender > >> > > > or > >> > > > > > >> simply > >> > > > > > >> > > add > >> > > > > > >> > > > a new one for log4j2? Reading the KIP and with its > >> current > >> > > > > title, > >> > > > > > >> it's > >> > > > > > >> > > not > >> > > > > > >> > > > entirely explicit. > >> > > > > > >> > > > > >> > > > > > >> > > > Oh, After re-reading it, I realized that this is not > >> > clear. > >> > > > Let > >> > > > > me > >> > > > > > >> > > clarify; > >> > > > > > >> > > > > >> > > > > > >> > > > 1. Provide a lo4j2 equivalent of traditional > >> > log4j-appender, > >> > > > > > >> > > > log4j2-appender. > >> > > > > > >> > > > 2. Migrate the modules depending on log4j-appender > >> (i.e., > >> > > > tools, > >> > > > > > >> trogdor, > >> > > > > > >> > > > shell) into log4j2-appender, removing log4j-appender > >> from > >> > > > > > >> dependencies. > >> > > > > > >> > > > 3. Entirely remove log4j-appender from the project > >> > > > dependencies, > >> > > > > > >> along > >> > > > > > >> > > with > >> > > > > > >> > > > log4j. > >> > > > > > >> > > > > >> > > > > > >> > > > I think log4j-appender may be published for every new > >> > > release > >> > > > > like > >> > > > > > >> > > before, > >> > > > > > >> > > > but the committee should make a decision on the > policy. > >> > > > > > >> > > > > >> > > > > > >> > > > > Under Rejected Alternative, the KIP states: "the > Kafka > >> > > > > appender > >> > > > > > >> > > provided > >> > > > > > >> > > > by log4j2 community stores log message in the Record > >> key". > >> > > > > Looking > >> > > > > > >> at the > >> > > > > > >> > > > code, it looks like the log message is stored in the > >> > Record > >> > > > > value: > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135 > >> > > > > > >> > > > Am I missing something? > >> > > > > > >> > > > > >> > > > > > >> > > > It's totally my fault; I confused it with another > >> > appender. > >> > > > The > >> > > > > > >> > > > compatibility problem in the logging-log4j2 Kafka > >> appender > >> > > is > >> > > > > not > >> > > > > > >> the > >> > > > > > >> > > > format but the configuration. logging-log4j2 Kafka > >> > appender > >> > > > > supports > >> > > > > > >> > > > `properties` configuration, which will be directly > used > >> to > >> > > > > > >> instantiate a > >> > > > > > >> > > > Kafka producer. However, log4j-appender has been > using > >> > > > > non-producer > >> > > > > > >> > > config > >> > > > > > >> > > > names like brokerList (=bootstrap.servers), > >> > requiredNumAcks > >> > > > > (=acks). > >> > > > > > >> > > > Instead, logging-log4j2 Kafka appender supports > >> > retryCount, > >> > > > > > >> > > > sendEventTimestamp. > >> > > > > > >> > > > > >> > > > > > >> > > > On second thought, using logging-log4j2 Kafka > appender > >> > > > > internally > >> > > > > > >> and > >> > > > > > >> > > > making log4j2-appender to focus on compatibility > facade > >> > only > >> > > > > would > >> > > > > > >> be a > >> > > > > > >> > > > better approach; As I described above, the goal of > this > >> > > module > >> > > > > is > >> > > > > > >> just > >> > > > > > >> > > > keeping the backward-compatibility, and (as you > pointed > >> > out) > >> > > > the > >> > > > > > >> current > >> > > > > > >> > > > implementation has little value. Since > >> > > > > > >> > > org.apache.logging.log4j:log4j-core > >> > > > > > >> > > > already includes Kafka appender, we can make use of > the > >> > > > 'proven > >> > > > > > >> wheel' > >> > > > > > >> > > > without adding more dependencies. I have not tried it > >> yet, > >> > > > but I > >> > > > > > >> think it > >> > > > > > >> > > > is well worth it. (One additional advantage of this > >> > approach > >> > > > is > >> > > > > > >> > > providing a > >> > > > > > >> > > > bridge to the users who hope to move from/into > >> > > logging-log4j2 > >> > > > > Kafka > >> > > > > > >> > > > appender.) > >> > > > > > >> > > > > >> > > > > > >> > > > > As the current log4j-appender is not even > deprecated > >> > yet, > >> > > in > >> > > > > > >> theory we > >> > > > > > >> > > > can't remove it till Kafka 4. If we want to speed up > the > >> > > > > process, I > >> > > > > > >> > > wonder > >> > > > > > >> > > > if the lack of documentation and a migration guide > could > >> > > help > >> > > > > us. > >> > > > > > >> What do > >> > > > > > >> > > > you think? > >> > > > > > >> > > > > >> > > > > > >> > > > In fact, this is what I am doing nowadays. While > working > >> > > with > >> > > > > > >> > > > log4j-appender, I found that despite a lack of > >> > > documentation, > >> > > > > > >> > > considerable > >> > > > > > >> > > > users are already using it[^1][^2][^3][^4][^5]. So, I > >> > think > >> > > > > > >> providing a > >> > > > > > >> > > > documentation to those who are already using > >> > log4j-appender > >> > > is > >> > > > > > >> > > > indispensable. It should include: > >> > > > > > >> > > > > >> > > > > > >> > > > - What is the difference between log4j-appender vs. > >> > > > > log4j2-appender. > >> > > > > > >> > > > - Which options are supported and deprecated. > >> > > > > > >> > > > - Exemplar configurations that show how to migrate. > >> > > > > > >> > > > > >> > > > > > >> > > > Here is the summary: > >> > > > > > >> > > > > >> > > > > > >> > > > 1. The goal of this proposal is to replace the > >> traditional > >> > > > > > >> log4j-appender > >> > > > > > >> > > > for compatibility concerns. But log4j-appender may be > >> > > > published > >> > > > > > >> after the > >> > > > > > >> > > > deprecation. > >> > > > > > >> > > > 2. As of present, the description about > logging-log4j2 > >> > Kafka > >> > > > > > >> appender is > >> > > > > > >> > > > entirely wrong. The problem is interface > compatibility, > >> > not > >> > > > > record > >> > > > > > >> > > format. > >> > > > > > >> > > > Focusing on the compatibility facade is a good > approach. > >> > > > > > >> > > > 3. A documentation focus on migration should be > >> provided. > >> > > > > > >> > > > > >> > > > > > >> > > > If you have any questions or suggestions, don't > hesitate > >> > to > >> > > > > tell me. > >> > > > > > >> > > Thanks > >> > > > > > >> > > > again for your comments! > >> > > > > > >> > > > > >> > > > > > >> > > > Best, > >> > > > > > >> > > > Dongjin > >> > > > > > >> > > > > >> > > > > > >> > > > [^1]: > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.cloudera.com/csa/1.2.0/monitoring/topics/csa-kafka-logging.html > >> > > > > > >> > > > [^2]: > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender > >> > > > > > >> > > > [^3]: > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://stackoverflow.com/questions/32402405/delay-in-kafka-log4j-appender > >> > > > > > >> > > > [^4]: > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://stackoverflow.com/questions/32301129/kafka-log4j-appender-not-sending-messages > >> > > > > > >> > > > [^5]: > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://stackoverflow.com/questions/35628706/kafka-log4j-appender-0-9-does-not-work > >> > > > > > >> > > > > >> > > > > > >> > > > On Mon, Nov 8, 2021 at 9:04 PM Mickael Maison < > >> > > > > > >> mickael.mai...@gmail.com> > >> > > > > > >> > > > wrote: > >> > > > > > >> > > > > >> > > > > > >> > > > > Hi Dongjin, > >> > > > > > >> > > > > > >> > > > > > >> > > > > Thanks for working on the update to log4j2, it's > >> > > > definitively > >> > > > > > >> > > > > something we should complete. > >> > > > > > >> > > > > I have a couple of comments: > >> > > > > > >> > > > > > >> > > > > > >> > > > > 1) Is the KIP proposing to replace the existing > >> > > > log4-appender > >> > > > > or > >> > > > > > >> > > > > simply add a new one for log4j2? Reading the KIP > and > >> > with > >> > > > its > >> > > > > > >> current > >> > > > > > >> > > > > title, it's not entirely explicit. For example I > don't > >> > > see a > >> > > > > > >> statement > >> > > > > > >> > > > > under the proposed changes section. The PR seems to > >> only > >> > > add > >> > > > > a new > >> > > > > > >> > > > > appender but the KIP mentions we want to fully > remove > >> > > > > > >> dependencies to > >> > > > > > >> > > > > log4j. > >> > > > > > >> > > > > > >> > > > > > >> > > > > 2) Under Rejected Alternative, the KIP states: "the > >> > Kafka > >> > > > > appender > >> > > > > > >> > > > > provided by log4j2 community stores log message in > the > >> > > > Record > >> > > > > > >> key". > >> > > > > > >> > > > > Looking at the code, it looks like the log message > is > >> > > stored > >> > > > > in > >> > > > > > >> the > >> > > > > > >> > > > > Record value: > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135 > >> > > > > > >> > > > > Am I missing something? > >> > > > > > >> > > > > Comparing it with the proposed new appender, apart > >> from > >> > > > their > >> > > > > > >> > > > > configuration format (hence the backwards > >> compatibility > >> > > > > issues), > >> > > > > > >> they > >> > > > > > >> > > > > both work pretty much the same way, so it's not > clear > >> it > >> > > > would > >> > > > > > >> add a > >> > > > > > >> > > > > ton a value. > >> > > > > > >> > > > > > >> > > > > > >> > > > > At a glance, _I've not extensively looked at it_, > it > >> > does > >> > > > not > >> > > > > look > >> > > > > > >> > > > > very hard to migrate to the appender from the > logging > >> > > team. > >> > > > I > >> > > > > was > >> > > > > > >> > > > > wondering if we should mention it in our > documentation > >> > > but I > >> > > > > was > >> > > > > > >> not > >> > > > > > >> > > > > able to find any references to the log4j-appender > in > >> the > >> > > > Kafka > >> > > > > > >> docs: > >> > > > > > >> > > > > > >> > > > > > https://github.com/apache/kafka-site/search?q=KafkaLog4jAppender > >> > > > > > >> > > > > > >> > > > > > >> > > > > As the current log4j-appender is not even > deprecated > >> > yet, > >> > > in > >> > > > > > >> theory we > >> > > > > > >> > > > > can't remove it till Kafka 4. If we want to speed > up > >> the > >> > > > > process, > >> > > > > > >> I > >> > > > > > >> > > > > wonder if the lack of documentation and a migration > >> > guide > >> > > > > could > >> > > > > > >> help > >> > > > > > >> > > > > us. What do you think? > >> > > > > > >> > > > > > >> > > > > > >> > > > > Thanks, > >> > > > > > >> > > > > Mickael > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, Jun 11, 2021 at 4:57 PM Boojapho O < > >> > > > > booja...@gmail.com> > >> > > > > > >> wrote: > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > Continuing to use log4j would leave several known > >> > > security > >> > > > > > >> > > > > vulnerabilities in Apache Kafka, including > >> > > > > > >> > > > > https://nvd.nist.gov/vuln/detail/CVE-2019-17571. > The > >> > > > Apache > >> > > > > > >> log4j > >> > > > > > >> > > team > >> > > > > > >> > > > > will not fix this vulnerability and is urging an > >> upgrade > >> > > to > >> > > > > > >> log4j2. > >> > > > > > >> > > See > >> > > > > > >> > > > > https://logging.apache.org/log4j/1.2/ for further > >> > > > > information. > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > This is desperately needed in Apache 3.0 to keep > the > >> > > > > software > >> > > > > > >> secure. > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > On 2021/05/26 12:31:20, Dongjin Lee < > >> > dong...@apache.org > >> > > > > >> > > > > wrote: > >> > > > > > >> > > > > > > CC'd the +1ers of KIP-653 with detailed > context: > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > When I submitted and got the approval of > KIP-653: > >> > > > Upgrade > >> > > > > > >> log4j to > >> > > > > > >> > > > > log4j2 > >> > > > > > >> > > > > > > < > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2 > >> > > > > > >> > > > > >, > >> > > > > > >> > > > > > > I thought the log4j2-appender should not be the > >> > scope > >> > > of > >> > > > > the > >> > > > > > >> work. > >> > > > > > >> > > But > >> > > > > > >> > > > > it > >> > > > > > >> > > > > > > was wrong. > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > Since the VerifiableLog4jAppender tool is built > >> upon > >> > > > > > >> > > log4j-appender, > >> > > > > > >> > > > > log4j > >> > > > > > >> > > > > > > 1.x artifact will co-exist with log4j2 > artifact in > >> > the > >> > > > > > >> classpath > >> > > > > > >> > > within > >> > > > > > >> > > > > > > this scheme. Since the log4j 1.x code is not > >> called > >> > > > > anymore, I > >> > > > > > >> > > thought > >> > > > > > >> > > > > it > >> > > > > > >> > > > > > > is not problematic but actually, it was not - > >> when I > >> > > > > started > >> > > > > > >> to > >> > > > > > >> > > > > provide a > >> > > > > > >> > > > > > > preview of KIP-653 > >> > > > > > >> > > > > > > < > >> > > > > > >> > >> > http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/ > >> > > > > > >> > > >, > >> > > > > > >> > > > > some > >> > > > > > >> > > > > > > users reported that sometimes slf4j fails to > find > >> > the > >> > > > > > >> appropriate > >> > > > > > >> > > > > binding > >> > > > > > >> > > > > > > within the classpath, resulting fail to append > the > >> > log > >> > > > > > >> message. > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > To resolve this problem, I subtly adjusted the > >> scope > >> > > of > >> > > > > the > >> > > > > > >> work; I > >> > > > > > >> > > > > > > excluded Tools and Trogdor from KIP-653 and > >> extended > >> > > > > KIP-719 > >> > > > > > >> to > >> > > > > > >> > > take > >> > > > > > >> > > > > care > >> > > > > > >> > > > > > > of them instead, along with providing > >> > log4j2-appender. > >> > > > It > >> > > > > is > >> > > > > > >> why > >> > > > > > >> > > the > >> > > > > > >> > > > > > > current WIP implementations include some > classpath > >> > > logic > >> > > > > in > >> > > > > > >> the > >> > > > > > >> > > shell > >> > > > > > >> > > > > > > script and *why KIP-653 only can't complete the > >> > log4j2 > >> > > > > > >> migration*. > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > I hope you will check this proposal out. > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > Best, > >> > > > > > >> > > > > > > Dongjin > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > On Tue, May 25, 2021 at 10:43 PM Dongjin Lee < > >> > > > > > >> dong...@apache.org> > >> > > > > > >> > > > > wrote: > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > Bumping up the discussion thread. > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > Recently, I updated the document of KIP-653: > >> > Upgrade > >> > > > > log4j > >> > > > > > >> to > >> > > > > > >> > > log4j2 > >> > > > > > >> > > > > > > > < > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2 > >> > > > > > >> > > > > >> > > > > > >> > > > > (accepted) > >> > > > > > >> > > > > > > > and KIP-719: Add Log4J2 Appender > >> > > > > > >> > > > > > > > < > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender > >> > > > > > >> > > > > >> > > > > > >> > > > > (under > >> > > > > > >> > > > > > > > discussion) reflecting the recent changes to > our > >> > > > > codebase. > >> > > > > > >> > > > > Especially: > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > 1. KIP-653 document > >> > > > > > >> > > > > > > > < > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2 > >> > > > > > >> > > > > >> > > > > > >> > > > > now > >> > > > > > >> > > > > > > > explains which modules will be migrated and > why. > >> > > > > > >> > > > > > > > 2. KIP-719 document > >> > > > > > >> > > > > > > > < > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender > >> > > > > > >> > > > > >> > > > > > >> > > > > now > >> > > > > > >> > > > > > > > explains not only the log4j2-appender plan > but > >> > also > >> > > > > > >> upgrading the > >> > > > > > >> > > > > omitted > >> > > > > > >> > > > > > > > modules in KIP-653 into log4j2. > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > As you can see here, those two KIPs are the > >> > > different > >> > > > > parts > >> > > > > > >> of > >> > > > > > >> > > the > >> > > > > > >> > > > > same > >> > > > > > >> > > > > > > > problem. I believe the community will have a > >> good > >> > > > grasp > >> > > > > on > >> > > > > > >> why > >> > > > > > >> > > both > >> > > > > > >> > > > > KIPs > >> > > > > > >> > > > > > > > are best if released altogether. > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > I will open the voting thread now, and please > >> > leave > >> > > a > >> > > > > vote > >> > > > > > >> if > >> > > > > > >> > > you are > >> > > > > > >> > > > > > > > interested in this issue. > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > Best, > >> > > > > > >> > > > > > > > Dongjin > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee < > >> > > > > > >> dong...@apache.org> > >> > > > > > >> > > > > wrote: > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > >> Hi Kafka dev, > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> I would like to start the discussion of > >> KIP-719: > >> > > Add > >> > > > > Log4J2 > >> > > > > > >> > > > > Appender. > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> All kinds of feedbacks are greatly > appreciated! > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> Best, > >> > > > > > >> > > > > > > >> Dongjin > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> -- > >> > > > > > >> > > > > > > >> *Dongjin Lee* > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> *A hitchhiker in the mathematical world.* > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > >> *github: <http://goog_969573159/> > >> > > > > github.com/dongjinleekr > >> > > > > > >> > > > > > > >> <https://github.com/dongjinleekr>keybase: > >> > > > > > >> > > > > https://keybase.io/dongjinleekr > >> > > > > > >> > > > > > > >> <https://keybase.io/dongjinleekr>linkedin: > >> > > > > > >> > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > >> > > > > > > >> <https://kr.linkedin.com/in/dongjinleekr > >> > > > >speakerdeck: > >> > > > > > >> > > > > speakerdeck.com/dongjin > >> > > > > > >> > > > > > > >> <https://speakerdeck.com/dongjin>* > >> > > > > > >> > > > > > > >> > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > -- > >> > > > > > >> > > > > > > > *Dongjin Lee* > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > *A hitchhiker in the mathematical world.* > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > *github: <http://goog_969573159/> > >> > > > > github.com/dongjinleekr > >> > > > > > >> > > > > > > > <https://github.com/dongjinleekr>keybase: > >> > > > > > >> > > > > https://keybase.io/dongjinleekr > >> > > > > > >> > > > > > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > > >> > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > >> > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > >> > > >speakerdeck: > >> > > > > > >> > > > > speakerdeck.com/dongjin > >> > > > > > >> > > > > > > > <https://speakerdeck.com/dongjin>* > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > -- > >> > > > > > >> > > > > > > *Dongjin Lee* > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > *A hitchhiker in the mathematical world.* > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > *github: <http://goog_969573159/> > >> > > > github.com/dongjinleekr > >> > > > > > >> > > > > > > <https://github.com/dongjinleekr>keybase: > >> > > > > > >> > > > > https://keybase.io/dongjinleekr > >> > > > > > >> > > > > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > > >> > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > >> > > > > > > <https://kr.linkedin.com/in/dongjinleekr > >> > >speakerdeck: > >> > > > > > >> > > > > speakerdeck.com/dongjin > >> > > > > > >> > > > > > > <https://speakerdeck.com/dongjin>* > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > -- > >> > > > > > >> > > > *Dongjin Lee* > >> > > > > > >> > > > > >> > > > > > >> > > > *A hitchhiker in the mathematical world.* > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > *github: <http://goog_969573159/> > >> github.com/dongjinleekr > >> > > > > > >> > > > <https://github.com/dongjinleekr>keybase: > >> > > > > > >> > > https://keybase.io/dongjinleekr > >> > > > > > >> > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > > >> > > kr.linkedin.com/in/dongjinleekr > >> > > > > > >> > > > <https://kr.linkedin.com/in/dongjinleekr > >speakerdeck: > >> > > > > > >> > > speakerdeck.com/dongjin > >> > > > > > >> > > > <https://speakerdeck.com/dongjin>* > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > -- > >> > > > > > >> > *Dongjin Lee* > >> > > > > > >> > > >> > > > > > >> > *A hitchhiker in the mathematical world.* > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > *github: <http://goog_969573159/> > github.com/dongjinleekr > >> > > > > > >> > <https://github.com/dongjinleekr>keybase: > >> > > > > > >> https://keybase.io/dongjinleekr > >> > > > > > >> > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > > >> kr.linkedin.com/in/dongjinleekr > >> > > > > > >> > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > > > > > >> speakerdeck.com/dongjin > >> > > > > > >> > <https://speakerdeck.com/dongjin>* > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > -- > >> > > > > > > *Dongjin Lee* > >> > > > > > > > >> > > > > > > *A hitchhiker in the mathematical world.* > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > >> > > > > > > <https://github.com/dongjinleekr>keybase: > >> > > > > https://keybase.io/dongjinleekr > >> > > > > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > > > > speakerdeck.com/dongjin > >> > > > > > > <https://speakerdeck.com/dongjin>* > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > *Dongjin Lee* > >> > > > > > > >> > > > > > *A hitchhiker in the mathematical world.* > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > >> > > > > > <https://github.com/dongjinleekr>keybase: > >> > > > > https://keybase.io/dongjinleekr > >> > > > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > > > kr.linkedin.com/in/dongjinleekr > >> > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > > > > speakerdeck.com/dongjin > >> > > > > > <https://speakerdeck.com/dongjin>* > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > *Dongjin Lee* > >> > > > > >> > > > *A hitchhiker in the mathematical world.* > >> > > > > >> > > > > >> > > > > >> > > > *github: <http://goog_969573159/>github.com/dongjinleekr > >> > > > <https://github.com/dongjinleekr>keybase: > >> > > https://keybase.io/dongjinleekr > >> > > > <https://keybase.io/dongjinleekr>linkedin: > >> > > kr.linkedin.com/in/dongjinleekr > >> > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > > > speakerdeck.com/dongjin > >> > > > <https://speakerdeck.com/dongjin>* > >> > > > > >> > > > >> > > > >> > > -- > >> > > ======================== > >> > > Okada Haruki > >> > > ocadar...@gmail.com > >> > > ======================== > >> > > > >> > > >> > > >> > -- > >> > *Dongjin Lee* > >> > > >> > *A hitchhiker in the mathematical world.* > >> > > >> > > >> > > >> > *github: <http://goog_969573159/>github.com/dongjinleekr > >> > <https://github.com/dongjinleekr>keybase: > >> https://keybase.io/dongjinleekr > >> > <https://keybase.io/dongjinleekr>linkedin: > >> kr.linkedin.com/in/dongjinleekr > >> > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> > speakerdeck.com/dongjin > >> > <https://speakerdeck.com/dongjin>* > >> > > >> > >> > >> -- > >> ======================== > >> Okada Haruki > >> ocadar...@gmail.com > >> ======================== > >> >
