I have directed it to the community, the PMC members. I am looking for a decision from the PMC members (including You) on whether we should continue as planned or not removing log4j2 form the minor release branch.
On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <ralph.go...@dslextreme.com> wrote: > Was this directed at me? I am not sure what decision you are referring to. > I stated my position in the email you replied to. > > Ralph > > > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo <fsz...@cloudera.com.INVALID> > wrote: > > > > With the new release, you can just replace the jars on the classpath as > we > > removed every code dependency and using slf4j. So you do not need to fork > > and change anything. > > > > Let me know your decision and we will continue with that. > > > > Regards, > > Ferenc > > > > On Mon, Nov 26, 2018 at 12:10 PM Apache <ralph.go...@dslextreme.com> > wrote: > > > >> As I said, I am fine with adding a notice to the release notes stating > the > >> logging will be changed in the next release regardless of its version > >> number. This way the revert only needs to happen on the release branch > for > >> this release. > >> > >> Ralph > >> > >>> On Nov 25, 2018, at 9:59 PM, Mike Percy <mpe...@apache.org> wrote: > >>> > >>> While the committer veto rules are well documented here > >>> <https://www.apache.org/foundation/voting.html#Veto>, and there is no > >>> mention of a time limit, I propose we shelve that discussion and work > on > >>> getting to consensus on what we as a community want to include in the > >> Flume > >>> 1.9 release. > >>> > >>> As far as I can tell, the decision we have to make is whether to > include > >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can either: > >>> > >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to > modify > >>> their log4j configuration files to use the different log4j2 XML format, > >>> with the procedure for doing so documented in the release notes. > >>> -or- > >>> 2) Defer that change to a future release where incompatible changes are > >>> expected, such as Flume 2.0. > >>> > >>> Also, maybe there are other options I haven't thought of...? > >>> > >>> I would like to get some input from more people on this matter. How do > >>> others feel about this? > >>> > >>> Thanks, > >>> Mike > >>> > >>> > >>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers < > ralph.go...@dslextreme.com> > >>> wrote: > >>> > >>>> I should also point out that the time to raise this was a year ago > when > >>>> the PR for FLUME-2050 was reviewed and committed. As it stands now I > >> could > >>>> be a jerk and vote -1 on the patch for FLUME-3296 with valid technical > >>>> grounds. If this was causing a true binary incompatibility I would > >> approve > >>>> reverting it in a heartbeat, but I just don’t see how having users > have > >> to > >>>> change logging configuration is “intolerable”, especially with the > known > >>>> security issues in Log4j 1, however unlikely user’s might be to > >> encounter > >>>> them. > >>>> > >>>> That said, I wouldn’t veto the revert on the release branch, but I > would > >>>> suggest that the release notes provide fair warning that the next > >> release > >>>> will upgrade the logging dependency. It would also be nice if > releases > >>>> could be more frequent than once a year. > >>>> > >>>> I would also like to say that I’m not doing this for the fun of it. My > >>>> company uses Flume for some of its most critical processing. We run > >>>> Veracode scans on all of our software and I expect Flume would be > >> flagged > >>>> if I hadn’t repacked it with Log4j 2. It also may not show up since > the > >> CVE > >>>> is marked against Log4j 2 since Log4j 1 is EOL, but the security > >> scanning > >>>> tools should be flagging that as well. > >>>> > >>>> FWIW I’ve had to hack or enhance a few Flume components to make it > work > >>>> for my needs but overall it works really, really well. I’d like to > >> commit > >>>> back some of the changes but the main one - Flume configuration - I > >> really > >>>> don’t like and need to redo the whole thing. > >>>> > >>>> Ralph > >>>> > >>>>> On Nov 24, 2018, at 5:11 PM, Mike Percy <mpe...@apache.org> wrote: > >>>>> > >>>>> I wasn't aware of this security issue. Do you have a link to the > >> details? > >>>>> > >>>>> There's no ASF requirement for package names; it's really just a Java > >>>>> language convention, and especially not enforced for compatibility > >> shims. > >>>>> Even Flume has some of those in the legacy sources. > >>>>> > >>>>> I understand that it's very difficult to provide a compatibility > layer. > >>>>> Maybe also a boring task. I'm just saying that without log4j1 > backwards > >>>>> compatibility provided by log4j2, there will have to be a really > >> critical > >>>>> reason to inflict this migration pain on Flume users -- something > that > >>>>> simply can't be tolerated, even in a minor release, like a new and > >> highly > >>>>> dangerous security bug. Without such a motivation, I don't see how > this > >>>>> incompatible dependency change can be justified. > >>>>> > >>>>> Mike > >>>>> > >>>>> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers < > >> ralph.go...@dslextreme.com> > >>>>> wrote: > >>>>> > >>>>>> > >>>>>>> On Nov 24, 2018, at 2:35 PM, Mike Percy <mpe...@apache.org> wrote: > >>>>>>> > >>>>>>> Flume has long had a policy of backwards compatibility with its own > >>>>>>> configuration files and people expect things to "just work" when > >>>>>> upgrading > >>>>>>> Flume. If log4j2 can't parse the log4j1 config file format then > it's > >> an > >>>>>>> incompatible upgrade and should not be done in a minor Flume > release. > >>>>>>> > >>>>>>> If log4j2 wants to be a drop-in replacement for log4j1 then by > >> default > >>>> it > >>>>>>> should find and parse the traditional log4j.properties config > files, > >> at > >>>>>>> least as a fallback, rather than force users to convert to the new > >> XML > >>>>>>> format before upgrading. > >>>>>> > >>>>>> That is simply not possible: > >>>>>> 1. Log4j 1 requires the use of fully qualified class names. The > >> package > >>>>>> names log4j 1 used didn’t conform to ASF naming guidelines and don’t > >>>> line > >>>>>> up with the package names used in Log4j2. > >>>>>> 2. Log4j 1 did not delineate between what was public and what was > >>>> private > >>>>>> so there is code all over the place mucking with the internals of > >> Log4j. > >>>>>> Log4j 2 implemented a compatibility layer by implementing the > classes > >>>> that > >>>>>> are being used but by and large they don’t actually do anything. > >>>>>> 3. The Appenders and Filters in Log4j 2 implement different > interfaces > >>>> and > >>>>>> cannot use components from Log4j 1. > >>>>>> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, > >>>> many > >>>>>> people didn’t use the Rolling File Appender that shipped with Log4j > >> but > >>>>>> used the one from Log4j extras. So it is hard to.know what Log4j 2 > >> would > >>>>>> have needed to be compatible with. > >>>>>> > >>>>>> Many organizations (including mine) have security requirements that > >> say > >>>>>> they must use software that is supported with security fixes. Log4j > 1 > >>>> has a > >>>>>> known security bug that will never be fixed as it reached > end-of-life > >> in > >>>>>> August of 2015. This means the use of Log4j 1 is not acceptable for > >> any > >>>>>> security conscious organization. > >>>>>> > >>>>>> While folks have brought up this issue from time to time most seem > to > >>>>>> adapt quite quickly to the change. Most of the issues are developers > >>>>>> wanting to know how to make something they were doing in Log4j 1 > work > >> in > >>>>>> Log4j 2. > >>>>>> > >>>>>> As you can imagine, since Log4j 2 has been GA now for 4 1/2 years > and > >>>>>> Log4j 1 has been EOL for over 3, making the configuration compatible > >>>> with > >>>>>> Log4j 1 is not a high priority. FWIW, there was an effort to do > that a > >>>> few > >>>>>> years ago, and while we were able to get some basic stuff to work > >>>> making it > >>>>>> work in a general way wasn’t possible. > >>>>>> > >>>>>> Ralph > >>>>>> > >>>>>>> > >>>>>>> Mike > >>>>>>> > >>>>>>> On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers < > >>>> ralph.go...@dslextreme.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> No, you are correct. However, requiring a change to logging > >>>>>> configuration > >>>>>>>> has never been considered a binary compatibility break in any > >> project > >>>> I > >>>>>>>> have ever worked on. > >>>>>>>> > >>>>>>>> Ralph > >>>>>>>> > >>>>>>>>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo > >>>> <fsz...@cloudera.com.INVALID > >>>>>>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> As you mentioned, you have the freedom to use Log4j 2 and the > same > >>>> time > >>>>>>>> we > >>>>>>>>> have to keep the out of the box experience the same in a minor > >>>> version. > >>>>>>>>> Users should be able to upgrade flume without changing any of > their > >>>>>>>>> configurations. > >>>>>>>>> If they have a log4j.properties (Log4j 1) then they would not be > >> able > >>>>>> to > >>>>>>>>> use it after the upgrade without changing it. > >>>>>>>>> > >>>>>>>>> Or am I missing a feature that would solve this case? > >>>>>>>>> > >>>>>>>>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers < > >>>>>> ralph.go...@dslextreme.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Also, please put details in the Jira issues. It is much easier > to > >>>> find > >>>>>>>> out > >>>>>>>>>> why something was done by searching Jira later on then searching > >> the > >>>>>>>>>> mailing list. > >>>>>>>>>> > >>>>>>>>>> Ralph > >>>>>>>>>> > >>>>>>>>>>> On Nov 23, 2018, at 9:47 AM, Ralph Goers < > >>>> ralph.go...@dslextreme.com > >>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I do not understand this at all. Log4j 2 provides runtime > >>>>>> compatibility > >>>>>>>>>> with Log4j 1. What is the problem that requires a revert? > >>>>>>>>>>> > >>>>>>>>>>> I have been running Flume with Log4j 2 since 1.6 so I don’t > >>>>>> understand > >>>>>>>>>> what the problem could possibly be. > >>>>>>>>>>> > >>>>>>>>>>> Ralph > >>>>>>>>>>> > >>>>>>>>>>>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo < > >> szabofe...@apache.org> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi everyone > >>>>>>>>>>>> > >>>>>>>>>>>> I am about to branch the 1.9 release from trunk. > >>>>>>>>>>>> > >>>>>>>>>>>> On the 1.9 branch we will revert the following breaking > changes: > >>>>>>>>>>>> - FLUME-2957. Remove Guava from our public API: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088 > >>>>>>>>>>>> - part of FLUME-2050. Upgrade to Log4j 2.10.0: > >>>>>>>>>>>> as the new release should work with the previous > configurations > >> we > >>>>>>>> have > >>>>>>>>>>>> to release it with log4j 1.x > >>>>>>>>>>>> For the log4j2 upgrade, we will provide a guide, how to > replace > >>>> the > >>>>>>>> jars > >>>>>>>>>>>> if users would like to start using it in the 1.9 release on > the > >>>> wiki > >>>>>>>>>> page. > >>>>>>>>>>>> > >>>>>>>>>>>> Because of these changes the first release candidate might be > >>>>>>>> postponed > >>>>>>>>>> to > >>>>>>>>>>>> Monday. > >>>>>>>>>>>> > >>>>>>>>>>>> Regards, > >>>>>>>>>>>> Ferenc Szabo > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com < > >>>>>>>> ema...@cloudera.com > >>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Ferenc, > >>>>>>>>>>>>> > >>>>>>>>>>>>> +1 > >>>>>>>>>>>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should > >> be > >>>>>> able > >>>>>>>>>> to > >>>>>>>>>>>>> finish it > >>>>>>>>>>>>> till the suggested deadline. > >>>>>>>>>>>>> I am also happy to do some reviews. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards > >>>>>>>>>>>>> Endre > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 2018/11/06 21:23:17, Ferenc Szabo <szabofe...@apache.org> > >>>>>> wrote: > >>>>>>>>>>>>>> Hello Flume Community, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1.8 was released about a year ago and since that quite a few > >> bug > >>>>>>>>>> fixes, > >>>>>>>>>>>>>> improvements, features and documentation were introduced. > >>>>>>>>>>>>>> I would like to propose to publish the next minor release of > >>>> Flume > >>>>>>>>>>>>>> to make these changes available to the users. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I would be more than happy to be the Release Manager with > the > >>>> help > >>>>>>>> of > >>>>>>>>>>>>>> Denes Arvay for anything that requires PMC access - if both > >> the > >>>>>>>>>> community > >>>>>>>>>>>>>> and he are > >>>>>>>>>>>>>> OK with it. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Among others the following changes will be included in the > >> next > >>>>>>>>>> release: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Fixed bugs: > >>>>>>>>>>>>>> - FLUME-3117 Application can be dead loop when call > >>>> System.exit() > >>>>>> in > >>>>>>>>>>>>>> methodconfigure > >>>>>>>>>>>>>> - FLUME-3237 Handling RuntimeExceptions coming from the JMS > >>>>>> provider > >>>>>>>>>> in > >>>>>>>>>>>>>> JMSSource > >>>>>>>>>>>>>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in > >> december > >>>>>>>>>>>>> correctly > >>>>>>>>>>>>>> - FLUME-3056 TestApplication hangs indefinitely > >>>>>>>>>>>>>> - FLUME-2976 Exception when JMS source tries to connect to a > >>>>>>>> Weblogic > >>>>>>>>>>>>>> server without authentication > >>>>>>>>>>>>>> - FLUME-3270 Close JMS resources in JMSMessageConsumer > >>>> constructor > >>>>>>>> in > >>>>>>>>>>>>> case > >>>>>>>>>>>>>> of failure > >>>>>>>>>>>>>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when > >> files > >>>>>> are > >>>>>>>>>>>>> being > >>>>>>>>>>>>>> deleted from the TAILDIR source > >>>>>>>>>>>>>> - FLUME-2894 Flume components should stop in the correct > order > >>>>>>>>>> (graceful > >>>>>>>>>>>>>> shutdown) > >>>>>>>>>>>>>> - FLUME-2973 Deadlock in hdfs sink > >>>>>>>>>>>>>> - FLUME-3278 Handling -D keystore parameters in Kafka > >> components > >>>>>>>>>>>>>> - FLUME-3265 Cannot set batch-size for > LoadBalancingRpcClient > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Improvements: > >>>>>>>>>>>>>> - FLUME-3186 Make asyncHbaseClient configuration parameters > >>>>>>>> available > >>>>>>>>>>>>> from > >>>>>>>>>>>>>> flume config > >>>>>>>>>>>>>> - FLUME-3239 Do not rename files in SpoolDirectorySource > >>>>>>>>>>>>>> - FLUME-3227 Add Rate Limiter to StressSource > >>>>>>>>>>>>>> - FLUME-2050 Upgrade to log4j2 (when GA) > >>>>>>>>>>>>>> - FLUME-3246 Validate flume configuration to prevent larger > >>>> source > >>>>>>>>>> batch > >>>>>>>>>>>>>> size than the channel transaction capacity > >>>>>>>>>>>>>> - FLUME-3050 add counters for error conditions and expose to > >>>>>> monitor > >>>>>>>>>> URL > >>>>>>>>>>>>>> - FLUME-3269 Support JSSE keystore/trustore -D system > >> properties > >>>>>>>>>>>>>> - FLUME-3223 Flume HDFS Sink should retry close prior to > >>>>>> performing > >>>>>>>> a > >>>>>>>>>>>>>> recoverLease > >>>>>>>>>>>>>> - FLUME-3276 Components supporting SSL/TLS should be able to > >>>>>> specify > >>>>>>>>>>>>> cipher > >>>>>>>>>>>>>> suite list > >>>>>>>>>>>>>> - FLUME-3275 Components supporting SSL/TLS should be able to > >>>>>> specify > >>>>>>>>>>>>>> protocol list > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> New Features: > >>>>>>>>>>>>>> - FLUME-3142 Adding HBase2 sink > >>>>>>>>>>>>>> - FLUME-2442 Need an alternative to providing clear text > >>>> passwords > >>>>>>>> in > >>>>>>>>>>>>> flume > >>>>>>>>>>>>>> config > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> There are 26 open tickets targeted for 1.9 in patch > available > >>>>>> state: > >>>>>>>>>>>>>> https://s.apache.org/flume-1.9-target-tickets > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We also have a number of open pull requests on Github: > >>>>>>>>>>>>>> https://github.com/apache/flume/pulls > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> There are a few tickets in progress that would be good to > >>>> include > >>>>>> in > >>>>>>>>>> the > >>>>>>>>>>>>>> release so > >>>>>>>>>>>>>> I would say that we focus on reviewing and testing them in > the > >>>>>> next > >>>>>>>> 2 > >>>>>>>>>>>>>> weeks. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I would like to propose to target the 23rd of November for > the > >>>>>> first > >>>>>>>>>>>>>> release candidate. > >>>>>>>>>>>>>> That would mean that the branch date would be a few days > >> before > >>>>>>>> that. > >>>>>>>>>>>>>> Any significant code change should get in by that date. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If nobody has any concerns then I am going to create an > >> umbrella > >>>>>>>>>> ticket > >>>>>>>>>>>>> to > >>>>>>>>>>>>>> track the release process. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Some movement can be expected in the related JIRA tickets. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Kind regards, > >>>>>>>>>>>>>> Ferenc > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >>>> > >> > >> > >> > > >
