Re: [ANNOUNCE] Apache Flume 1.7.0 released

2016-10-19 Thread Hari Shreedharan 
Thanks Donat and Mike! Great work!

On Wed, Oct 19, 2016 at 3:06 AM, Mike Percy  wrote:
> Thank you for all your hard work RMing this release Donat and getting it to
> the finish line. It was my pleasure to help out!
>
> Best,
> Mike
>
> On Tue, Oct 18, 2016 at 1:32 PM, Balazs Donat Bessenyei > wrote:
>
>> And thank you, Mike Percy for the mentoring and tremendous amounts of
>> assistance with the release!
>>
>> On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
>>  wrote:
>> > Thank you all very much who participated and helped with the release!
>> >
>> >
>> > Donat
>> >
>> >
>> > On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
>> >> Woot! Congrats everyone!
>> >>
>> >> Thanks Donat for working so hard to get this version of Flume out the
>> door!
>> >>
>> >> Best,
>> >> Mike
>> >>
>> >>
>> >> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <
>> bes...@apache.org>
>> >> wrote:
>> >>>
>> >>> The Apache Flume team is pleased to announce the release of Flume
>> >>> version 1.7.0.
>> >>>
>> >>> Flume is a distributed, reliable, and available service for efficiently
>> >>> collecting, aggregating, and moving large amounts of log data.
>> >>>
>> >>> This release can be downloaded from the Flume download page at:
>> >>> http://flume.apache.org/download.html
>> >>>
>> >>> The change log and documentation are available on the 1.7.0 release
>> page:
>> >>> http://flume.apache.org/releases/1.7.0.html
>> >>>
>> >>> Your help and feedback is more than welcome. For more information on
>> how
>> >>> to report problems and to get involved, visit the project website at
>> >>> http://flume.apache.org/
>> >>>
>> >>> The Apache Flume Team
>> >>
>> >>
>>

Re: Review Request 51244: FLUME-2171: Add Interceptor to remove headers from event

2016-10-19 Thread Attila Simon 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review153273
---




w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java
 (lines 119 - 138)


wasRemoved is not needed here. if removedHeaders list in not empty then 
there were a removal.


Otherwise it looks good to me.

- Attila Simon


On Aug. 24, 2016, 9:08 p.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> ---
> 
> (Updated Aug. 24, 2016, 9:08 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> I found Flume OG's decorators to handle event headers useful and some to be 
> missing from Flume NG. More specifically, we could have an interceptor to 
> remove headers from an event.
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java
>  fe341e9 
>   c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6 
>   
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java
>  PRE-CREATION 
>   
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51244/diff/
> 
> 
> Testing
> ---
> 
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run 
> successfully
> 
> I've used this config for manual testing:
> 
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 
> a1.sources.r1.channels = c1
> 
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1
> a1.channels.c1.transactionCapacity = 1
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 80
> 
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
> 
> 
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
> 
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
> 
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 51244: FLUME-2171: Add Interceptor to remove headers from event

2016-10-19 Thread Attila Simon 


> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java,
> >  line 139
> > 
> >
> > Please don't log out the whole Event for each removed header. Also 
> > please collect all the headers which were removed and print them out as a 
> > single log entry per Event.
> 
> Balázs Donát Bessenyei wrote:
> I think there is a benefit to keeping the event in the log, but I changed 
> the code to collect the headers for logging
> 
> If you think the whole event logging is a blocking issue, please re-open

Fair enough. Then could you please consider wrapping the logging of event with 
a org.apache.flume.conf.LogPrivacyUtil#allowLogRawData() check?


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146665
---


On Aug. 24, 2016, 9:08 p.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> ---
> 
> (Updated Aug. 24, 2016, 9:08 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> I found Flume OG's decorators to handle event headers useful and some to be 
> missing from Flume NG. More specifically, we could have an interceptor to 
> remove headers from an event.
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java
>  fe341e9 
>   c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6 
>   
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java
>  PRE-CREATION 
>   
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51244/diff/
> 
> 
> Testing
> ---
> 
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run 
> successfully
> 
> I've used this config for manual testing:
> 
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 
> a1.sources.r1.channels = c1
> 
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1
> a1.channels.c1.transactionCapacity = 1
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 80
> 
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
> 
> 
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
> 
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
> 
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52550: FLUME-2989: Review Request

2016-10-19 Thread Umesh Chaudhary 


> On Oct. 18, 2016, 9:44 a.m., Balázs Donát Bessenyei wrote:
> > flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java,
> >  line 769
> > 
> >
> > Do you think this Thread.sleep could be avoided?
> > 
> > If not, do you think you can change it in a way that it will be 
> > not-so-prone to flakiness?
> > (Maybe something like at https://reviews.apache.org/r/49025/diff/5#3 )

Need Thread.sleep to spawn the HTTPMetricsServer before making HTTP GET request 
for metrics. 

Reduced the sleep time for 10ms so that it would not show flakiness.


On Oct. 18, 2016, 9:44 a.m., Umesh Chaudhary wrote:
> > Thank you for the patch!
> > 
> > Can you please rebase it on trunk so that it applies cleanly?
> > 
> > Also, please reformat the code to pass checkstyle checks.

Rebased in on trunk and fixed the checkstyle error.


- Umesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52550/#review153062
---


On Oct. 19, 2016, 10:17 a.m., Umesh Chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52550/
> ---
> 
> (Updated Oct. 19, 2016, 10:17 a.m.)
> 
> 
> Review request for Flume, Balázs Donát Bessenyei, Jeff Holoman, and Mike 
> Percy.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added appropriate function calls to capture eventTakeAttemptCount and 
> eventPutAttemptCount in KafkaChannel
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
>  47c0634 
>   
> flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
>  276fee1 
> 
> Diff: https://reviews.apache.org/r/52550/diff/
> 
> 
> Testing
> ---
> 
> # mvn  -DfailIfNoTests=false 
> -Dtest=org.apache.flume.channel.kafka.TestKafkaChannel test
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 04:17 min
> [INFO] Finished at: 2016-10-19T15:31:38+05:30
> [INFO] Final Memory: 98M/1210M
> [INFO] 
> 
> 
> # mvn -pl flume-ng-channels -Drat.skip=true test
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1.108 s
> [INFO] Finished at: 2016-10-19T15:44:52+05:30
> [INFO] Final Memory: 23M/309M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Umesh Chaudhary
> 
>

Re: Review Request 52550: FLUME-2989: Review Request

2016-10-19 Thread Umesh Chaudhary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52550/
---

(Updated Oct. 19, 2016, 10:17 a.m.)


Review request for Flume, Balázs Donát Bessenyei, Jeff Holoman, and Mike Percy.


Repository: flume-git


Description
---

Added appropriate function calls to capture eventTakeAttemptCount and 
eventPutAttemptCount in KafkaChannel


Diffs (updated)
-

  
flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
 47c0634 
  
flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
 276fee1 

Diff: https://reviews.apache.org/r/52550/diff/


Testing (updated)
---

# mvn  -DfailIfNoTests=false 
-Dtest=org.apache.flume.channel.kafka.TestKafkaChannel test

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 04:17 min
[INFO] Finished at: 2016-10-19T15:31:38+05:30
[INFO] Final Memory: 98M/1210M
[INFO] 

# mvn -pl flume-ng-channels -Drat.skip=true test

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1.108 s
[INFO] Finished at: 2016-10-19T15:44:52+05:30
[INFO] Final Memory: 23M/309M
[INFO] 


Thanks,

Umesh Chaudhary

Re: [ANNOUNCE] Apache Flume 1.7.0 released

2016-10-19 Thread Mike Percy 
Thank you for all your hard work RMing this release Donat and getting it to
the finish line. It was my pleasure to help out!

Best,
Mike

On Tue, Oct 18, 2016 at 1:32 PM, Balazs Donat Bessenyei  wrote:

> And thank you, Mike Percy for the mentoring and tremendous amounts of
> assistance with the release!
>
> On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
>  wrote:
> > Thank you all very much who participated and helped with the release!
> >
> >
> > Donat
> >
> >
> > On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
> >> Woot! Congrats everyone!
> >>
> >> Thanks Donat for working so hard to get this version of Flume out the
> door!
> >>
> >> Best,
> >> Mike
> >>
> >>
> >> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <
> bes...@apache.org>
> >> wrote:
> >>>
> >>> The Apache Flume team is pleased to announce the release of Flume
> >>> version 1.7.0.
> >>>
> >>> Flume is a distributed, reliable, and available service for efficiently
> >>> collecting, aggregating, and moving large amounts of log data.
> >>>
> >>> This release can be downloaded from the Flume download page at:
> >>> http://flume.apache.org/download.html
> >>>
> >>> The change log and documentation are available on the 1.7.0 release
> page:
> >>> http://flume.apache.org/releases/1.7.0.html
> >>>
> >>> Your help and feedback is more than welcome. For more information on
> how
> >>> to report problems and to get involved, visit the project website at
> >>> http://flume.apache.org/
> >>>
> >>> The Apache Flume Team
> >>
> >>
>