I wonder, is there anything blocking us from releasing 2.0 next instead of 1.x?
Donat On Mon, Nov 26, 2018 at 6:27 PM Mike Percy <mpe...@apache.org> wrote: > > If only the PMC would be affected by this decision, we could have had this > discussion on the PMC private list. But this decision impacts everybody > that uses Flume. So let's hear from anybody who cares about this, including > committers, contributors, and users on whether they are okay with switching > to log4j2 in a minor release version, knowing that they will need to change > their config files when they upgrade Flume. > > Ferenc, it seems like we will have to ship one or the other with the binary > artifacts at release time. It seems to me that we still have to make a > choice about the built and shipped default, even if both could work at > runtime, right? > > Thanks, > Mike > > On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo <fsz...@cloudera.com.invalid> > wrote: > > > I have directed it to the community, the PMC members. > > I am looking for a decision from the PMC members (including You) on whether > > we should continue as planned or not removing log4j2 form the minor release > > branch. > > > > On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <ralph.go...@dslextreme.com> > > wrote: > > > > > Was this directed at me? I am not sure what decision you are referring > > to. > > > I stated my position in the email you replied to. > > > > > > Ralph > > > > > > > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo <fsz...@cloudera.com.INVALID > > > > > > wrote: > > > > > > > > With the new release, you can just replace the jars on the classpath as > > > we > > > > removed every code dependency and using slf4j. So you do not need to > > fork > > > > and change anything. > > > > > > > > Let me know your decision and we will continue with that. > > > > > > > > Regards, > > > > Ferenc > > > > > > > > On Mon, Nov 26, 2018 at 12:10 PM Apache <ralph.go...@dslextreme.com> > > > wrote: > > > > > > > >> As I said, I am fine with adding a notice to the release notes stating > > > the > > > >> logging will be changed in the next release regardless of its version > > > >> number. This way the revert only needs to happen on the release branch > > > for > > > >> this release. > > > >> > > > >> Ralph > > > >> > > > >>> On Nov 25, 2018, at 9:59 PM, Mike Percy <mpe...@apache.org> wrote: > > > >>> > > > >>> While the committer veto rules are well documented here > > > >>> <https://www.apache.org/foundation/voting.html#Veto>, and there is > > no > > > >>> mention of a time limit, I propose we shelve that discussion and work > > > on > > > >>> getting to consensus on what we as a community want to include in the > > > >> Flume > > > >>> 1.9 release. > > > >>> > > > >>> As far as I can tell, the decision we have to make is whether to > > > include > > > >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can > > either: > > > >>> > > > >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to > > > modify > > > >>> their log4j configuration files to use the different log4j2 XML > > format, > > > >>> with the procedure for doing so documented in the release notes. > > > >>> -or- > > > >>> 2) Defer that change to a future release where incompatible changes > > are > > > >>> expected, such as Flume 2.0. > > > >>> > > > >>> Also, maybe there are other options I haven't thought of...? > > > >>> > > > >>> I would like to get some input from more people on this matter. How > > do > > > >>> others feel about this? > > > >>> > > > >>> Thanks, > > > >>> Mike > > > >>> > > > >>> > > > >>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers < > > > ralph.go...@dslextreme.com> > > > >>> wrote: > > > >>> > > > >>>> I should also point out that the time to raise this was a year ago > > > when > > > >>>> the PR for FLUME-2050 was reviewed and committed. As it stands now I > > > >> could > > > >>>> be a jerk and vote -1 on the patch for FLUME-3296 with valid > > technical > > > >>>> grounds. If this was causing a true binary incompatibility I would > > > >> approve > > > >>>> reverting it in a heartbeat, but I just don’t see how having users > > > have > > > >> to > > > >>>> change logging configuration is “intolerable”, especially with the > > > known > > > >>>> security issues in Log4j 1, however unlikely user’s might be to > > > >> encounter > > > >>>> them. > > > >>>> > > > >>>> That said, I wouldn’t veto the revert on the release branch, but I > > > would > > > >>>> suggest that the release notes provide fair warning that the next > > > >> release > > > >>>> will upgrade the logging dependency. It would also be nice if > > > releases > > > >>>> could be more frequent than once a year. > > > >>>> > > > >>>> I would also like to say that I’m not doing this for the fun of it. > > My > > > >>>> company uses Flume for some of its most critical processing. We run > > > >>>> Veracode scans on all of our software and I expect Flume would be > > > >> flagged > > > >>>> if I hadn’t repacked it with Log4j 2. It also may not show up since > > > the > > > >> CVE > > > >>>> is marked against Log4j 2 since Log4j 1 is EOL, but the security > > > >> scanning > > > >>>> tools should be flagging that as well. > > > >>>> > > > >>>> FWIW I’ve had to hack or enhance a few Flume components to make it > > > work > > > >>>> for my needs but overall it works really, really well. I’d like to > > > >> commit > > > >>>> back some of the changes but the main one - Flume configuration - I > > > >> really > > > >>>> don’t like and need to redo the whole thing. > > > >>>> > > > >>>> Ralph > > > >>>> > > > >>>>> On Nov 24, 2018, at 5:11 PM, Mike Percy <mpe...@apache.org> wrote: > > > >>>>> > > > >>>>> I wasn't aware of this security issue. Do you have a link to the > > > >> details? > > > >>>>> > > > >>>>> There's no ASF requirement for package names; it's really just a > > Java > > > >>>>> language convention, and especially not enforced for compatibility > > > >> shims. > > > >>>>> Even Flume has some of those in the legacy sources. > > > >>>>> > > > >>>>> I understand that it's very difficult to provide a compatibility > > > layer. > > > >>>>> Maybe also a boring task. I'm just saying that without log4j1 > > > backwards > > > >>>>> compatibility provided by log4j2, there will have to be a really > > > >> critical > > > >>>>> reason to inflict this migration pain on Flume users -- something > > > that > > > >>>>> simply can't be tolerated, even in a minor release, like a new and > > > >> highly > > > >>>>> dangerous security bug. Without such a motivation, I don't see how > > > this > > > >>>>> incompatible dependency change can be justified. > > > >>>>> > > > >>>>> Mike > > > >>>>> > > > >>>>> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers < > > > >> ralph.go...@dslextreme.com> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> > > > >>>>>>> On Nov 24, 2018, at 2:35 PM, Mike Percy <mpe...@apache.org> > > wrote: > > > >>>>>>> > > > >>>>>>> Flume has long had a policy of backwards compatibility with its > > own > > > >>>>>>> configuration files and people expect things to "just work" when > > > >>>>>> upgrading > > > >>>>>>> Flume. If log4j2 can't parse the log4j1 config file format then > > > it's > > > >> an > > > >>>>>>> incompatible upgrade and should not be done in a minor Flume > > > release. > > > >>>>>>> > > > >>>>>>> If log4j2 wants to be a drop-in replacement for log4j1 then by > > > >> default > > > >>>> it > > > >>>>>>> should find and parse the traditional log4j.properties config > > > files, > > > >> at > > > >>>>>>> least as a fallback, rather than force users to convert to the > > new > > > >> XML > > > >>>>>>> format before upgrading. > > > >>>>>> > > > >>>>>> That is simply not possible: > > > >>>>>> 1. Log4j 1 requires the use of fully qualified class names. The > > > >> package > > > >>>>>> names log4j 1 used didn’t conform to ASF naming guidelines and > > don’t > > > >>>> line > > > >>>>>> up with the package names used in Log4j2. > > > >>>>>> 2. Log4j 1 did not delineate between what was public and what was > > > >>>> private > > > >>>>>> so there is code all over the place mucking with the internals of > > > >> Log4j. > > > >>>>>> Log4j 2 implemented a compatibility layer by implementing the > > > classes > > > >>>> that > > > >>>>>> are being used but by and large they don’t actually do anything. > > > >>>>>> 3. The Appenders and Filters in Log4j 2 implement different > > > interfaces > > > >>>> and > > > >>>>>> cannot use components from Log4j 1. > > > >>>>>> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In > > fact, > > > >>>> many > > > >>>>>> people didn’t use the Rolling File Appender that shipped with > > Log4j > > > >> but > > > >>>>>> used the one from Log4j extras. So it is hard to.know what Log4j 2 > > > >> would > > > >>>>>> have needed to be compatible with. > > > >>>>>> > > > >>>>>> Many organizations (including mine) have security requirements > > that > > > >> say > > > >>>>>> they must use software that is supported with security fixes. > > Log4j > > > 1 > > > >>>> has a > > > >>>>>> known security bug that will never be fixed as it reached > > > end-of-life > > > >> in > > > >>>>>> August of 2015. This means the use of Log4j 1 is not acceptable > > for > > > >> any > > > >>>>>> security conscious organization. > > > >>>>>> > > > >>>>>> While folks have brought up this issue from time to time most seem > > > to > > > >>>>>> adapt quite quickly to the change. Most of the issues are > > developers > > > >>>>>> wanting to know how to make something they were doing in Log4j 1 > > > work > > > >> in > > > >>>>>> Log4j 2. > > > >>>>>> > > > >>>>>> As you can imagine, since Log4j 2 has been GA now for 4 1/2 years > > > and > > > >>>>>> Log4j 1 has been EOL for over 3, making the configuration > > compatible > > > >>>> with > > > >>>>>> Log4j 1 is not a high priority. FWIW, there was an effort to do > > > that a > > > >>>> few > > > >>>>>> years ago, and while we were able to get some basic stuff to work > > > >>>> making it > > > >>>>>> work in a general way wasn’t possible. > > > >>>>>> > > > >>>>>> Ralph > > > >>>>>> > > > >>>>>>> > > > >>>>>>> Mike > > > >>>>>>> > > > >>>>>>> On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers < > > > >>>> ralph.go...@dslextreme.com> > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> No, you are correct. However, requiring a change to logging > > > >>>>>> configuration > > > >>>>>>>> has never been considered a binary compatibility break in any > > > >> project > > > >>>> I > > > >>>>>>>> have ever worked on. > > > >>>>>>>> > > > >>>>>>>> Ralph > > > >>>>>>>> > > > >>>>>>>>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo > > > >>>> <fsz...@cloudera.com.INVALID > > > >>>>>>> > > > >>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> As you mentioned, you have the freedom to use Log4j 2 and the > > > same > > > >>>> time > > > >>>>>>>> we > > > >>>>>>>>> have to keep the out of the box experience the same in a minor > > > >>>> version. > > > >>>>>>>>> Users should be able to upgrade flume without changing any of > > > their > > > >>>>>>>>> configurations. > > > >>>>>>>>> If they have a log4j.properties (Log4j 1) then they would not > > be > > > >> able > > > >>>>>> to > > > >>>>>>>>> use it after the upgrade without changing it. > > > >>>>>>>>> > > > >>>>>>>>> Or am I missing a feature that would solve this case? > > > >>>>>>>>> > > > >>>>>>>>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers < > > > >>>>>> ralph.go...@dslextreme.com> > > > >>>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Also, please put details in the Jira issues. It is much easier > > > to > > > >>>> find > > > >>>>>>>> out > > > >>>>>>>>>> why something was done by searching Jira later on then > > searching > > > >> the > > > >>>>>>>>>> mailing list. > > > >>>>>>>>>> > > > >>>>>>>>>> Ralph > > > >>>>>>>>>> > > > >>>>>>>>>>> On Nov 23, 2018, at 9:47 AM, Ralph Goers < > > > >>>> ralph.go...@dslextreme.com > > > >>>>>>> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> I do not understand this at all. Log4j 2 provides runtime > > > >>>>>> compatibility > > > >>>>>>>>>> with Log4j 1. What is the problem that requires a revert? > > > >>>>>>>>>>> > > > >>>>>>>>>>> I have been running Flume with Log4j 2 since 1.6 so I don’t > > > >>>>>> understand > > > >>>>>>>>>> what the problem could possibly be. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Ralph > > > >>>>>>>>>>> > > > >>>>>>>>>>>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo < > > > >> szabofe...@apache.org> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hi everyone > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I am about to branch the 1.9 release from trunk. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On the 1.9 branch we will revert the following breaking > > > changes: > > > >>>>>>>>>>>> - FLUME-2957. Remove Guava from our public API: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > >>>> > > > >> > > > > > https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088 > > > >>>>>>>>>>>> - part of FLUME-2050. Upgrade to Log4j 2.10.0: > > > >>>>>>>>>>>> as the new release should work with the previous > > > configurations > > > >> we > > > >>>>>>>> have > > > >>>>>>>>>>>> to release it with log4j 1.x > > > >>>>>>>>>>>> For the log4j2 upgrade, we will provide a guide, how to > > > replace > > > >>>> the > > > >>>>>>>> jars > > > >>>>>>>>>>>> if users would like to start using it in the 1.9 release on > > > the > > > >>>> wiki > > > >>>>>>>>>> page. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Because of these changes the first release candidate might > > be > > > >>>>>>>> postponed > > > >>>>>>>>>> to > > > >>>>>>>>>>>> Monday. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Regards, > > > >>>>>>>>>>>> Ferenc Szabo > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com < > > > >>>>>>>> ema...@cloudera.com > > > >>>>>>>>>>> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi Ferenc, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> +1 > > > >>>>>>>>>>>>> I am working on FLUME-3281 Update to Kafka 2.0 client, > > should > > > >> be > > > >>>>>> able > > > >>>>>>>>>> to > > > >>>>>>>>>>>>> finish it > > > >>>>>>>>>>>>> till the suggested deadline. > > > >>>>>>>>>>>>> I am also happy to do some reviews. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Regards > > > >>>>>>>>>>>>> Endre > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On 2018/11/06 21:23:17, Ferenc Szabo < > > szabofe...@apache.org> > > > >>>>>> wrote: > > > >>>>>>>>>>>>>> Hello Flume Community, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> 1.8 was released about a year ago and since that quite a > > few > > > >> bug > > > >>>>>>>>>> fixes, > > > >>>>>>>>>>>>>> improvements, features and documentation were introduced. > > > >>>>>>>>>>>>>> I would like to propose to publish the next minor release > > of > > > >>>> Flume > > > >>>>>>>>>>>>>> to make these changes available to the users. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I would be more than happy to be the Release Manager with > > > the > > > >>>> help > > > >>>>>>>> of > > > >>>>>>>>>>>>>> Denes Arvay for anything that requires PMC access - if > > both > > > >> the > > > >>>>>>>>>> community > > > >>>>>>>>>>>>>> and he are > > > >>>>>>>>>>>>>> OK with it. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Among others the following changes will be included in the > > > >> next > > > >>>>>>>>>> release: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Fixed bugs: > > > >>>>>>>>>>>>>> - FLUME-3117 Application can be dead loop when call > > > >>>> System.exit() > > > >>>>>> in > > > >>>>>>>>>>>>>> methodconfigure > > > >>>>>>>>>>>>>> - FLUME-3237 Handling RuntimeExceptions coming from the > > JMS > > > >>>>>> provider > > > >>>>>>>>>> in > > > >>>>>>>>>>>>>> JMSSource > > > >>>>>>>>>>>>>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in > > > >> december > > > >>>>>>>>>>>>> correctly > > > >>>>>>>>>>>>>> - FLUME-3056 TestApplication hangs indefinitely > > > >>>>>>>>>>>>>> - FLUME-2976 Exception when JMS source tries to connect > > to a > > > >>>>>>>> Weblogic > > > >>>>>>>>>>>>>> server without authentication > > > >>>>>>>>>>>>>> - FLUME-3270 Close JMS resources in JMSMessageConsumer > > > >>>> constructor > > > >>>>>>>> in > > > >>>>>>>>>>>>> case > > > >>>>>>>>>>>>>> of failure > > > >>>>>>>>>>>>>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when > > > >> files > > > >>>>>> are > > > >>>>>>>>>>>>> being > > > >>>>>>>>>>>>>> deleted from the TAILDIR source > > > >>>>>>>>>>>>>> - FLUME-2894 Flume components should stop in the correct > > > order > > > >>>>>>>>>> (graceful > > > >>>>>>>>>>>>>> shutdown) > > > >>>>>>>>>>>>>> - FLUME-2973 Deadlock in hdfs sink > > > >>>>>>>>>>>>>> - FLUME-3278 Handling -D keystore parameters in Kafka > > > >> components > > > >>>>>>>>>>>>>> - FLUME-3265 Cannot set batch-size for > > > LoadBalancingRpcClient > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Improvements: > > > >>>>>>>>>>>>>> - FLUME-3186 Make asyncHbaseClient configuration > > parameters > > > >>>>>>>> available > > > >>>>>>>>>>>>> from > > > >>>>>>>>>>>>>> flume config > > > >>>>>>>>>>>>>> - FLUME-3239 Do not rename files in SpoolDirectorySource > > > >>>>>>>>>>>>>> - FLUME-3227 Add Rate Limiter to StressSource > > > >>>>>>>>>>>>>> - FLUME-2050 Upgrade to log4j2 (when GA) > > > >>>>>>>>>>>>>> - FLUME-3246 Validate flume configuration to prevent > > larger > > > >>>> source > > > >>>>>>>>>> batch > > > >>>>>>>>>>>>>> size than the channel transaction capacity > > > >>>>>>>>>>>>>> - FLUME-3050 add counters for error conditions and expose > > to > > > >>>>>> monitor > > > >>>>>>>>>> URL > > > >>>>>>>>>>>>>> - FLUME-3269 Support JSSE keystore/trustore -D system > > > >> properties > > > >>>>>>>>>>>>>> - FLUME-3223 Flume HDFS Sink should retry close prior to > > > >>>>>> performing > > > >>>>>>>> a > > > >>>>>>>>>>>>>> recoverLease > > > >>>>>>>>>>>>>> - FLUME-3276 Components supporting SSL/TLS should be able > > to > > > >>>>>> specify > > > >>>>>>>>>>>>> cipher > > > >>>>>>>>>>>>>> suite list > > > >>>>>>>>>>>>>> - FLUME-3275 Components supporting SSL/TLS should be able > > to > > > >>>>>> specify > > > >>>>>>>>>>>>>> protocol list > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> New Features: > > > >>>>>>>>>>>>>> - FLUME-3142 Adding HBase2 sink > > > >>>>>>>>>>>>>> - FLUME-2442 Need an alternative to providing clear text > > > >>>> passwords > > > >>>>>>>> in > > > >>>>>>>>>>>>> flume > > > >>>>>>>>>>>>>> config > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> There are 26 open tickets targeted for 1.9 in patch > > > available > > > >>>>>> state: > > > >>>>>>>>>>>>>> https://s.apache.org/flume-1.9-target-tickets > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> We also have a number of open pull requests on Github: > > > >>>>>>>>>>>>>> https://github.com/apache/flume/pulls > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> There are a few tickets in progress that would be good to > > > >>>> include > > > >>>>>> in > > > >>>>>>>>>> the > > > >>>>>>>>>>>>>> release so > > > >>>>>>>>>>>>>> I would say that we focus on reviewing and testing them in > > > the > > > >>>>>> next > > > >>>>>>>> 2 > > > >>>>>>>>>>>>>> weeks. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I would like to propose to target the 23rd of November for > > > the > > > >>>>>> first > > > >>>>>>>>>>>>>> release candidate. > > > >>>>>>>>>>>>>> That would mean that the branch date would be a few days > > > >> before > > > >>>>>>>> that. > > > >>>>>>>>>>>>>> Any significant code change should get in by that date. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> If nobody has any concerns then I am going to create an > > > >> umbrella > > > >>>>>>>>>> ticket > > > >>>>>>>>>>>>> to > > > >>>>>>>>>>>>>> track the release process. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Some movement can be expected in the related JIRA tickets. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Kind regards, > > > >>>>>>>>>>>>>> Ferenc > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >>>> > > > >> > > > >> > > > >> > > > > > > > > > > >
