I wasn't aware of this security issue. Do you have a link to the details? There's no ASF requirement for package names; it's really just a Java language convention, and especially not enforced for compatibility shims. Even Flume has some of those in the legacy sources.
I understand that it's very difficult to provide a compatibility layer. Maybe also a boring task. I'm just saying that without log4j1 backwards compatibility provided by log4j2, there will have to be a really critical reason to inflict this migration pain on Flume users -- something that simply can't be tolerated, even in a minor release, like a new and highly dangerous security bug. Without such a motivation, I don't see how this incompatible dependency change can be justified. Mike On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers <ralph.go...@dslextreme.com> wrote: > > > On Nov 24, 2018, at 2:35 PM, Mike Percy <mpe...@apache.org> wrote: > > > > Flume has long had a policy of backwards compatibility with its own > > configuration files and people expect things to "just work" when > upgrading > > Flume. If log4j2 can't parse the log4j1 config file format then it's an > > incompatible upgrade and should not be done in a minor Flume release. > > > > If log4j2 wants to be a drop-in replacement for log4j1 then by default it > > should find and parse the traditional log4j.properties config files, at > > least as a fallback, rather than force users to convert to the new XML > > format before upgrading. > > That is simply not possible: > 1. Log4j 1 requires the use of fully qualified class names. The package > names log4j 1 used didn’t conform to ASF naming guidelines and don’t line > up with the package names used in Log4j2. > 2. Log4j 1 did not delineate between what was public and what was private > so there is code all over the place mucking with the internals of Log4j. > Log4j 2 implemented a compatibility layer by implementing the classes that > are being used but by and large they don’t actually do anything. > 3. The Appenders and Filters in Log4j 2 implement different interfaces and > cannot use components from Log4j 1. > 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, many > people didn’t use the Rolling File Appender that shipped with Log4j but > used the one from Log4j extras. So it is hard to.know what Log4j 2 would > have needed to be compatible with. > > Many organizations (including mine) have security requirements that say > they must use software that is supported with security fixes. Log4j 1 has a > known security bug that will never be fixed as it reached end-of-life in > August of 2015. This means the use of Log4j 1 is not acceptable for any > security conscious organization. > > While folks have brought up this issue from time to time most seem to > adapt quite quickly to the change. Most of the issues are developers > wanting to know how to make something they were doing in Log4j 1 work in > Log4j 2. > > As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and > Log4j 1 has been EOL for over 3, making the configuration compatible with > Log4j 1 is not a high priority. FWIW, there was an effort to do that a few > years ago, and while we were able to get some basic stuff to work making it > work in a general way wasn’t possible. > > Ralph > > > > > Mike > > > > On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers <ralph.go...@dslextreme.com> > > wrote: > > > >> No, you are correct. However, requiring a change to logging > configuration > >> has never been considered a binary compatibility break in any project I > >> have ever worked on. > >> > >> Ralph > >> > >>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo <fsz...@cloudera.com.INVALID > > > >> wrote: > >>> > >>> As you mentioned, you have the freedom to use Log4j 2 and the same time > >> we > >>> have to keep the out of the box experience the same in a minor version. > >>> Users should be able to upgrade flume without changing any of their > >>> configurations. > >>> If they have a log4j.properties (Log4j 1) then they would not be able > to > >>> use it after the upgrade without changing it. > >>> > >>> Or am I missing a feature that would solve this case? > >>> > >>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers < > ralph.go...@dslextreme.com> > >>> wrote: > >>> > >>>> Also, please put details in the Jira issues. It is much easier to find > >> out > >>>> why something was done by searching Jira later on then searching the > >>>> mailing list. > >>>> > >>>> Ralph > >>>> > >>>>> On Nov 23, 2018, at 9:47 AM, Ralph Goers <ralph.go...@dslextreme.com > > > >>>> wrote: > >>>>> > >>>>> I do not understand this at all. Log4j 2 provides runtime > compatibility > >>>> with Log4j 1. What is the problem that requires a revert? > >>>>> > >>>>> I have been running Flume with Log4j 2 since 1.6 so I don’t > understand > >>>> what the problem could possibly be. > >>>>> > >>>>> Ralph > >>>>> > >>>>>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo <szabofe...@apache.org> > >>>> wrote: > >>>>>> > >>>>>> Hi everyone > >>>>>> > >>>>>> I am about to branch the 1.9 release from trunk. > >>>>>> > >>>>>> On the 1.9 branch we will revert the following breaking changes: > >>>>>> - FLUME-2957. Remove Guava from our public API: > >>>>>> > >>>>>> > >>>> > >> > https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088 > >>>>>> - part of FLUME-2050. Upgrade to Log4j 2.10.0: > >>>>>> as the new release should work with the previous configurations we > >> have > >>>>>> to release it with log4j 1.x > >>>>>> For the log4j2 upgrade, we will provide a guide, how to replace the > >> jars > >>>>>> if users would like to start using it in the 1.9 release on the wiki > >>>> page. > >>>>>> > >>>>>> Because of these changes the first release candidate might be > >> postponed > >>>> to > >>>>>> Monday. > >>>>>> > >>>>>> Regards, > >>>>>> Ferenc Szabo > >>>>>> > >>>>>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com < > >> ema...@cloudera.com > >>>>> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Ferenc, > >>>>>>> > >>>>>>> +1 > >>>>>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be > able > >>>> to > >>>>>>> finish it > >>>>>>> till the suggested deadline. > >>>>>>> I am also happy to do some reviews. > >>>>>>> > >>>>>>> Regards > >>>>>>> Endre > >>>>>>> > >>>>>>> On 2018/11/06 21:23:17, Ferenc Szabo <szabofe...@apache.org> > wrote: > >>>>>>>> Hello Flume Community, > >>>>>>>> > >>>>>>>> 1.8 was released about a year ago and since that quite a few bug > >>>> fixes, > >>>>>>>> improvements, features and documentation were introduced. > >>>>>>>> I would like to propose to publish the next minor release of Flume > >>>>>>>> to make these changes available to the users. > >>>>>>>> > >>>>>>>> I would be more than happy to be the Release Manager with the help > >> of > >>>>>>>> Denes Arvay for anything that requires PMC access - if both the > >>>> community > >>>>>>>> and he are > >>>>>>>> OK with it. > >>>>>>>> > >>>>>>>> Among others the following changes will be included in the next > >>>> release: > >>>>>>>> > >>>>>>>> Fixed bugs: > >>>>>>>> - FLUME-3117 Application can be dead loop when call System.exit() > in > >>>>>>>> methodconfigure > >>>>>>>> - FLUME-3237 Handling RuntimeExceptions coming from the JMS > provider > >>>> in > >>>>>>>> JMSSource > >>>>>>>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december > >>>>>>> correctly > >>>>>>>> - FLUME-3056 TestApplication hangs indefinitely > >>>>>>>> - FLUME-2976 Exception when JMS source tries to connect to a > >> Weblogic > >>>>>>>> server without authentication > >>>>>>>> - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor > >> in > >>>>>>> case > >>>>>>>> of failure > >>>>>>>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when files > are > >>>>>>> being > >>>>>>>> deleted from the TAILDIR source > >>>>>>>> - FLUME-2894 Flume components should stop in the correct order > >>>> (graceful > >>>>>>>> shutdown) > >>>>>>>> - FLUME-2973 Deadlock in hdfs sink > >>>>>>>> - FLUME-3278 Handling -D keystore parameters in Kafka components > >>>>>>>> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient > >>>>>>>> > >>>>>>>> > >>>>>>>> Improvements: > >>>>>>>> - FLUME-3186 Make asyncHbaseClient configuration parameters > >> available > >>>>>>> from > >>>>>>>> flume config > >>>>>>>> - FLUME-3239 Do not rename files in SpoolDirectorySource > >>>>>>>> - FLUME-3227 Add Rate Limiter to StressSource > >>>>>>>> - FLUME-2050 Upgrade to log4j2 (when GA) > >>>>>>>> - FLUME-3246 Validate flume configuration to prevent larger source > >>>> batch > >>>>>>>> size than the channel transaction capacity > >>>>>>>> - FLUME-3050 add counters for error conditions and expose to > monitor > >>>> URL > >>>>>>>> - FLUME-3269 Support JSSE keystore/trustore -D system properties > >>>>>>>> - FLUME-3223 Flume HDFS Sink should retry close prior to > performing > >> a > >>>>>>>> recoverLease > >>>>>>>> - FLUME-3276 Components supporting SSL/TLS should be able to > specify > >>>>>>> cipher > >>>>>>>> suite list > >>>>>>>> - FLUME-3275 Components supporting SSL/TLS should be able to > specify > >>>>>>>> protocol list > >>>>>>>> > >>>>>>>> > >>>>>>>> New Features: > >>>>>>>> - FLUME-3142 Adding HBase2 sink > >>>>>>>> - FLUME-2442 Need an alternative to providing clear text passwords > >> in > >>>>>>> flume > >>>>>>>> config > >>>>>>>> > >>>>>>>> > >>>>>>>> There are 26 open tickets targeted for 1.9 in patch available > state: > >>>>>>>> https://s.apache.org/flume-1.9-target-tickets > >>>>>>>> > >>>>>>>> We also have a number of open pull requests on Github: > >>>>>>>> https://github.com/apache/flume/pulls > >>>>>>>> > >>>>>>>> There are a few tickets in progress that would be good to include > in > >>>> the > >>>>>>>> release so > >>>>>>>> I would say that we focus on reviewing and testing them in the > next > >> 2 > >>>>>>>> weeks. > >>>>>>>> > >>>>>>>> I would like to propose to target the 23rd of November for the > first > >>>>>>>> release candidate. > >>>>>>>> That would mean that the branch date would be a few days before > >> that. > >>>>>>>> Any significant code change should get in by that date. > >>>>>>>> > >>>>>>>> If nobody has any concerns then I am going to create an umbrella > >>>> ticket > >>>>>>> to > >>>>>>>> track the release process. > >>>>>>>> > >>>>>>>> Some movement can be expected in the related JIRA tickets. > >>>>>>>> > >>>>>>>> Kind regards, > >>>>>>>> Ferenc > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >>>> > >>>> > >> > >> > >> > > >
