Re: Issue : Flume performance

[Ashish]

Missed this thread. No direct solution comes to my mind for this

Couple of things, since the degradation is occurring only with few instances.

1. Find out what is different on the machines seeing degradation from
the others (including OS configurations)
2. Is it the degradation random or occurring on specific hosts

Put me in this situation and I would do following

1. Try to look out for a Pattern of degradation, similar hosts, some
event storm or could be anything
2. If you have monitoring system, see what's happening around during
the degradation, heavy GC activity.

Once you get a similar pattern, perhaps debugging and solving it would
be easier.

On Mon, Apr 6, 2015 at 12:20 AM, ajeet singh  wrote:
> -- Forwarded message --
> From: ajeet singh 
> Date: Mon, Apr 6, 2015 at 12:03 AM
> Subject: Issue : Flume performance
> To: u...@flume.apache.org
>
>
> I am using flume for logging events from aws servers to KAFKA. I am using
> logback as flume source, and kafka-sink as sink. In servers we are getting
> 3000 events/ per seconds, and there are almost 30 servers running and one
> central kafka. At each server flume process is running which will be
> writing to central kafka.
>  I am getting some strange situations like at some servers
> flume is able for process the event with rate in which it's getting  the
> event but on some servers some time it get slow ( like it's getting event
> with rate 3k / sec but transferring events to kafka with rate 1Kv per sec).
> This way 2K /per sec events are getting added to flume channel, and after
> 3-4 hours channel overflow starts. I compared both type of servers where
> flume is working fine, and where flume is slow, all parameters ( load, cpu
> utilization, memory usages and disk writes) are almost same.  What can be
> reason for this strange behavior ??
>
> Thanks
> Ajeet
>
>
>
> --
> AJEET PRATAP SINGH



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: I want to be a contributor for apache flume-ng and I have a source here

[Ashish]

How is it different from
http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source?

More details on Getting involved here http://flume.apache.org/getinvolved.html

On Thu, Apr 9, 2015 at 6:46 AM, 青年才俊 <719259...@qq.com> wrote:
> Hi,dev team
>
>  as the topic of the mail
>
>  I write a source——file monitor source
>
>  it monitors a file and check the modified to get a file stream to record all 
> the modified content,and it will record the position in disk file,although 
> the flume application has been killed,OK,it record the position,so it knows 
> which line should be read next time when it starts.
>
>  I have a test on it ,it can sink all the checked contents without lost.
>
>  thank you!
>
>  here is my github repo:
>  https://github.com/cwtree/flume-filemonitor-source



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

I want to be a contributor for apache flume-ng and I have a source here

[????????]

Hi,dev team
  
 as the topic of the mail
  
 I write a sourcefile monitor source
  
 it monitors a file and check the modified to get a file stream to record all 
the modified content,and it will record the position in disk file,although the 
flume application has been killed,OK,it record the position,so it knows which 
line should be read next time when it starts.
  
 I have a test on it ,it can sink all the checked contents without lost.
  
 thank you!
  
 here is my github repo:
 https://github.com/cwtree/flume-filemonitor-source

Fwd: Need help with Custom Flume Interceptors

[Maheedhar Gunturu]

I am trying to build a small prototype of FLUME using custom interceptors.

I am looking to parse rsyslog using Custom Flume Regex interceptor and
input the values into various columns in one single column Family in hbase.

Here is a sample set of log messages that I have.

Mar 20 07:42:54 node46 ntpd[31266]: Deleting interface #4 em1,
fe80::862b:2bff:fefb:9065#123, interface stats: received=0, sent=0,
dropped=0, active_time=7995120 secs


As a next step I would like to select a few columns which can be input into
the respective columns in the column family.


One reference that I found is this article but it is very vague in the
description
https://questforthought.wordpress.com/2014/01/13/using-flume-interceptor-multiplexing/


Can someone help.

- Vanguard_space

[jira] [Updated] (FLUME-2595) Add option to checkpoint on file channel shutdown

[Roshan Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik updated FLUME-2595:
---
Attachment: FLUME-2595.v3.patch

[~hshreedharan] this patch fixes all UT failures in TestLog. Got a clean run on 
all FC UTs as well.

> Add option to checkpoint on file channel shutdown
> -
>
> Key: FLUME-2595
> URL: https://issues.apache.org/jira/browse/FLUME-2595
> Project: Flume
>  Issue Type: Bug
>  Components: File Channel
>Affects Versions: v1.5.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: channel
> Attachments: FLUME-2595.patch, FLUME-2595.v2.patch, 
> FLUME-2595.v3.patch
>
>
> Restarting a File channel based agent can take very long, a few hours to 
> evens days in some cases. The reason being the slow replay performance.
> By having the file channel checkpoint on shutdown, this long replay can 
> avoided as there will be nothing to replay on startup.
> A small hit (a few seconds perhaps) will be observed on shutdown to perform 
> the checkpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flume performance measurements

[Arvind Prabhakar]

Done. Please let me know if you run into any issues.

Regards,
Arvind

On Wed, Apr 8, 2015 at 3:58 PM, Roshan Naik  wrote:

> roshan_naik is my login to cwiki.apache.org
>
>
>
>
> On 4/8/15 3:55 PM, "Arvind Prabhakar"  wrote:
>
> >Added Hari to the wiki.
> >
> >Roshan, I could not look you up on the wiki users, can you please tell me
> >your username? If you don't have one yet, please register and let me know.
> >
> >Regards,
> >Arvind Prabhakar
> >
> >On Wed, Apr 8, 2015 at 3:26 PM, Roshan Naik 
> >wrote:
> >
> >> Arvind,
> >>   Please do let me know once  you have granted me permission to the
> >>wiki.
> >> -roshan
> >>
> >> From: Hari Shreedharan  >> hshreedha...@cloudera.com>>
> >> Date: Thursday, April 2, 2015 3:06 PM
> >> To: Roshan Naik mailto:ros...@hortonworks.com>>
> >> Cc: "dev@flume.apache.org" <
> >> dev@flume.apache.org>
> >> Subject: Re: Flume performance measurements
> >>
> >> Arvind - please could you grant Roshan access to the wiki.
> >>
> >> Thanks,
> >> Hari
> >>
> >>
> >>
> >> On Thu, Apr 2, 2015 at 3:04 PM, Roshan Naik  >> > wrote:
> >>
> >> Could u grant me write access to wiki ?
> >> username: roshannaik
> >>
> >>
> >>
> >> On 4/2/15 2:53 PM, "Hari Shreedharan"   >> hshreedha...@cloudera.com>> wrote:
> >>
> >> >Roshan,
> >> >
> >> >
> >> >
> >> >
> >> >Could you update the performance measurements page on our wiki with
> >>this
> >> >info? That would be more useful to reference.
> >> >
> >> >
> >> >
> >> >
> >> >Thanks, Hari
> >> >
> >> >On Thu, Apr 2, 2015 at 2:34 PM, Roshan Naik  >> >
> >> >wrote:
> >> >
> >> >> Sample Flume v1.4 Measurements for reference:
> >> >> Here are some sample measurements taken with a single agent and 500
> >> >>byte events.
> >> >> Cluster Config: 20-node Hadoop cluster (1 name node and 19 data
> >>nodes).
> >> >> Machine Config: 24 cores - Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM.
> >> >> 1. File channel with HDFS Sink (Sequence File):
> >> >> Source: 4 x Exec Source, 100k batchSize
> >> >> HDFS Sink Batch size: 500,000
> >> >> Channel: File
> >> >> Number of data dirs: 8
> >> >> Events/Sec
> >> >> Sink Count
> >> >> 1 data dirs
> >> >> 2 data dirs
> >> >> 4 data dirs
> >> >> 6 data dirs
> >> >> 8 data dirs
> >> >> 10 data dirs
> >> >> 1
> >> >> 14.3 k
> >> >> 2
> >> >> 21.9 k
> >> >> 4
> >> >> 35.8 k
> >> >> 8
> >> >> 24.8 k
> >> >> 43.8 k
> >> >> 72.5 k
> >> >> 77 k
> >> >> 78.6 k
> >> >> 76.6 k
> >> >> 10
> >> >> 58 k
> >> >> 12
> >> >> 49.3 k
> >> >> 49 k
> >> >> Was looking for sweet spot in perf. So did not take measurements for
> >> >>all data points on grid. Only too for the ones that made sense. For
> >> >>example: when perf dropped by adding more sinks, did not take more
> >> >>measurements for those rows.
> >> >> 2. HDFS Sink:
> >> >> Channel: Memory
> >> >> # of HDFS
> >> >> Sinks
> >> >> Snappy
> >> >> BatchSz:1.2mill
> >> >> Snappy
> >> >> BatchSz:1.4mill
> >> >> Sequence File
> >> >> BatchSz:1.2mill
> >> >> 1
> >> >> 34.3 k
> >> >> 33 k
> >> >> 33 k
> >> >> 2
> >> >> 71 k
> >> >> 75 k
> >> >> 69 k
> >> >> 4
> >> >> 141 k
> >> >> 145 k
> >> >> 141 k
> >> >> 8
> >> >> 271 k
> >> >> 273 k
> >> >> 251 k
> >> >> 12
> >> >> 382 k
> >> >> 380 k
> >> >> 370 k
> >> >> 16
> >> >> 478 k
> >> >> 538 k
> >> >> 486 k
> >> >> Some simple observations :
> >> >> * increasing number of dataDirs helps FC perf even on single disk
> >> >>systems
> >> >> * Increasing number of sinks helps
> >> >> * Max throughput observed was about 538k events/sec for HDFS sink
> >> >>which is approx 240MB/s
> >>
> >>
> >>
>
>

[jira] [Commented] (FLUME-2663) Address Build warnings of duplicate dependencies listed

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486279#comment-14486279
 ] 

Roshan Naik commented on FLUME-2663:


just removed duplicate dependency listings

> Address Build warnings of duplicate dependencies listed
> ---
>
> Key: FLUME-2663
> URL: https://issues.apache.org/jira/browse/FLUME-2663
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: v1.5.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2663.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2663) Address Build warnings of duplicate dependencies listed

[Roshan Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik updated FLUME-2663:
---
Attachment: FLUME-2663.patch

Uploading patch

> Address Build warnings of duplicate dependencies listed
> ---
>
> Key: FLUME-2663
> URL: https://issues.apache.org/jira/browse/FLUME-2663
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: v1.5.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2663.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flume performance measurements

[Roshan Naik]

roshan_naik is my login to cwiki.apache.org




On 4/8/15 3:55 PM, "Arvind Prabhakar"  wrote:

>Added Hari to the wiki.
>
>Roshan, I could not look you up on the wiki users, can you please tell me
>your username? If you don't have one yet, please register and let me know.
>
>Regards,
>Arvind Prabhakar
>
>On Wed, Apr 8, 2015 at 3:26 PM, Roshan Naik 
>wrote:
>
>> Arvind,
>>   Please do let me know once  you have granted me permission to the
>>wiki.
>> -roshan
>>
>> From: Hari Shreedharan > hshreedha...@cloudera.com>>
>> Date: Thursday, April 2, 2015 3:06 PM
>> To: Roshan Naik mailto:ros...@hortonworks.com>>
>> Cc: "dev@flume.apache.org" <
>> dev@flume.apache.org>
>> Subject: Re: Flume performance measurements
>>
>> Arvind - please could you grant Roshan access to the wiki.
>>
>> Thanks,
>> Hari
>>
>>
>>
>> On Thu, Apr 2, 2015 at 3:04 PM, Roshan Naik > > wrote:
>>
>> Could u grant me write access to wiki ?
>> username: roshannaik
>>
>>
>>
>> On 4/2/15 2:53 PM, "Hari Shreedharan" > hshreedha...@cloudera.com>> wrote:
>>
>> >Roshan,
>> >
>> >
>> >
>> >
>> >Could you update the performance measurements page on our wiki with
>>this
>> >info? That would be more useful to reference.
>> >
>> >
>> >
>> >
>> >Thanks, Hari
>> >
>> >On Thu, Apr 2, 2015 at 2:34 PM, Roshan Naik > >
>> >wrote:
>> >
>> >> Sample Flume v1.4 Measurements for reference:
>> >> Here are some sample measurements taken with a single agent and 500
>> >>byte events.
>> >> Cluster Config: 20-node Hadoop cluster (1 name node and 19 data
>>nodes).
>> >> Machine Config: 24 cores - Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM.
>> >> 1. File channel with HDFS Sink (Sequence File):
>> >> Source: 4 x Exec Source, 100k batchSize
>> >> HDFS Sink Batch size: 500,000
>> >> Channel: File
>> >> Number of data dirs: 8
>> >> Events/Sec
>> >> Sink Count
>> >> 1 data dirs
>> >> 2 data dirs
>> >> 4 data dirs
>> >> 6 data dirs
>> >> 8 data dirs
>> >> 10 data dirs
>> >> 1
>> >> 14.3 k
>> >> 2
>> >> 21.9 k
>> >> 4
>> >> 35.8 k
>> >> 8
>> >> 24.8 k
>> >> 43.8 k
>> >> 72.5 k
>> >> 77 k
>> >> 78.6 k
>> >> 76.6 k
>> >> 10
>> >> 58 k
>> >> 12
>> >> 49.3 k
>> >> 49 k
>> >> Was looking for sweet spot in perf. So did not take measurements for
>> >>all data points on grid. Only too for the ones that made sense. For
>> >>example: when perf dropped by adding more sinks, did not take more
>> >>measurements for those rows.
>> >> 2. HDFS Sink:
>> >> Channel: Memory
>> >> # of HDFS
>> >> Sinks
>> >> Snappy
>> >> BatchSz:1.2mill
>> >> Snappy
>> >> BatchSz:1.4mill
>> >> Sequence File
>> >> BatchSz:1.2mill
>> >> 1
>> >> 34.3 k
>> >> 33 k
>> >> 33 k
>> >> 2
>> >> 71 k
>> >> 75 k
>> >> 69 k
>> >> 4
>> >> 141 k
>> >> 145 k
>> >> 141 k
>> >> 8
>> >> 271 k
>> >> 273 k
>> >> 251 k
>> >> 12
>> >> 382 k
>> >> 380 k
>> >> 370 k
>> >> 16
>> >> 478 k
>> >> 538 k
>> >> 486 k
>> >> Some simple observations :
>> >> * increasing number of dataDirs helps FC perf even on single disk
>> >>systems
>> >> * Increasing number of sinks helps
>> >> * Max throughput observed was about 538k events/sec for HDFS sink
>> >>which is approx 240MB/s
>>
>>
>>

Re: Flume performance measurements

[Arvind Prabhakar]

Added Hari to the wiki.

Roshan, I could not look you up on the wiki users, can you please tell me
your username? If you don't have one yet, please register and let me know.

Regards,
Arvind Prabhakar

On Wed, Apr 8, 2015 at 3:26 PM, Roshan Naik  wrote:

> Arvind,
>   Please do let me know once  you have granted me permission to the wiki.
> -roshan
>
> From: Hari Shreedharan  hshreedha...@cloudera.com>>
> Date: Thursday, April 2, 2015 3:06 PM
> To: Roshan Naik mailto:ros...@hortonworks.com>>
> Cc: "dev@flume.apache.org" <
> dev@flume.apache.org>
> Subject: Re: Flume performance measurements
>
> Arvind - please could you grant Roshan access to the wiki.
>
> Thanks,
> Hari
>
>
>
> On Thu, Apr 2, 2015 at 3:04 PM, Roshan Naik  > wrote:
>
> Could u grant me write access to wiki ?
> username: roshannaik
>
>
>
> On 4/2/15 2:53 PM, "Hari Shreedharan"  hshreedha...@cloudera.com>> wrote:
>
> >Roshan,
> >
> >
> >
> >
> >Could you update the performance measurements page on our wiki with this
> >info? That would be more useful to reference.
> >
> >
> >
> >
> >Thanks, Hari
> >
> >On Thu, Apr 2, 2015 at 2:34 PM, Roshan Naik  >
> >wrote:
> >
> >> Sample Flume v1.4 Measurements for reference:
> >> Here are some sample measurements taken with a single agent and 500
> >>byte events.
> >> Cluster Config: 20-node Hadoop cluster (1 name node and 19 data nodes).
> >> Machine Config: 24 cores - Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM.
> >> 1. File channel with HDFS Sink (Sequence File):
> >> Source: 4 x Exec Source, 100k batchSize
> >> HDFS Sink Batch size: 500,000
> >> Channel: File
> >> Number of data dirs: 8
> >> Events/Sec
> >> Sink Count
> >> 1 data dirs
> >> 2 data dirs
> >> 4 data dirs
> >> 6 data dirs
> >> 8 data dirs
> >> 10 data dirs
> >> 1
> >> 14.3 k
> >> 2
> >> 21.9 k
> >> 4
> >> 35.8 k
> >> 8
> >> 24.8 k
> >> 43.8 k
> >> 72.5 k
> >> 77 k
> >> 78.6 k
> >> 76.6 k
> >> 10
> >> 58 k
> >> 12
> >> 49.3 k
> >> 49 k
> >> Was looking for sweet spot in perf. So did not take measurements for
> >>all data points on grid. Only too for the ones that made sense. For
> >>example: when perf dropped by adding more sinks, did not take more
> >>measurements for those rows.
> >> 2. HDFS Sink:
> >> Channel: Memory
> >> # of HDFS
> >> Sinks
> >> Snappy
> >> BatchSz:1.2mill
> >> Snappy
> >> BatchSz:1.4mill
> >> Sequence File
> >> BatchSz:1.2mill
> >> 1
> >> 34.3 k
> >> 33 k
> >> 33 k
> >> 2
> >> 71 k
> >> 75 k
> >> 69 k
> >> 4
> >> 141 k
> >> 145 k
> >> 141 k
> >> 8
> >> 271 k
> >> 273 k
> >> 251 k
> >> 12
> >> 382 k
> >> 380 k
> >> 370 k
> >> 16
> >> 478 k
> >> 538 k
> >> 486 k
> >> Some simple observations :
> >> * increasing number of dataDirs helps FC perf even on single disk
> >>systems
> >> * Increasing number of sinks helps
> >> * Max throughput observed was about 538k events/sec for HDFS sink
> >>which is approx 240MB/s
>
>
>

[jira] [Updated] (FLUME-2662) Upgrade to Commons-IO 2.4

[Roshan Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik updated FLUME-2662:
---
Attachment: FLUME-2662.patch

Uploading patch

> Upgrade to Commons-IO 2.4
> -
>
> Key: FLUME-2662
> URL: https://issues.apache.org/jira/browse/FLUME-2662
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: v1.5.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: Dependencies
> Attachments: FLUME-2662.patch
>
>
> Hadoop 2.7 is now switching to apache-commons-io v2.4. Hbase 1.0 is also 
> using commons-io v2.4.
> Flume is currently at 2.1.
> Flume runs into issues like this when tests run against them:
> testSequenceFile(org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem)  Time 
> elapsed: 77 sec  <<< ERROR!
> java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
> at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:854)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:69)
> at 
> org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem.testSequenceFile(TestUseRawLocalFileSystem.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> I am planning to submit a patch to upgrade commons-io to 2.4.  Just wanted to 
> be cautious and check if we have witnessed has been any issue in the past 
> when upgrading apache commons libraries.
> Based on what I see here:
> http://commons.apache.org/proper/commons-io/upgradeto2_4.html  and
> http://commons.apache.org/proper/commons-io/upgradeto2_2.html
> Commons-IO 2.4 is binary compat with 2.2 which is in turn binary compat
> with 2.1.
> There is what they call a "rare" case of source incompat as described in
> https://issues.apache.org/jira/browse/IO-318
> Doesnt look like we are affected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flume performance measurements

[Roshan Naik]

Arvind,
  Please do let me know once  you have granted me permission to the wiki.
-roshan

From: Hari Shreedharan 
mailto:hshreedha...@cloudera.com>>
Date: Thursday, April 2, 2015 3:06 PM
To: Roshan Naik mailto:ros...@hortonworks.com>>
Cc: "dev@flume.apache.org" 
mailto:dev@flume.apache.org>>
Subject: Re: Flume performance measurements

Arvind - please could you grant Roshan access to the wiki.

Thanks,
Hari



On Thu, Apr 2, 2015 at 3:04 PM, Roshan Naik 
mailto:ros...@hortonworks.com>> wrote:

Could u grant me write access to wiki ?
username: roshannaik



On 4/2/15 2:53 PM, "Hari Shreedharan" 
mailto:hshreedha...@cloudera.com>> wrote:

>Roshan,
>
>
>
>
>Could you update the performance measurements page on our wiki with this
>info? That would be more useful to reference.
>
>
>
>
>Thanks, Hari
>
>On Thu, Apr 2, 2015 at 2:34 PM, Roshan Naik 
>mailto:ros...@hortonworks.com>>
>wrote:
>
>> Sample Flume v1.4 Measurements for reference:
>> Here are some sample measurements taken with a single agent and 500
>>byte events.
>> Cluster Config: 20-node Hadoop cluster (1 name node and 19 data nodes).
>> Machine Config: 24 cores - Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM.
>> 1. File channel with HDFS Sink (Sequence File):
>> Source: 4 x Exec Source, 100k batchSize
>> HDFS Sink Batch size: 500,000
>> Channel: File
>> Number of data dirs: 8
>> Events/Sec
>> Sink Count
>> 1 data dirs
>> 2 data dirs
>> 4 data dirs
>> 6 data dirs
>> 8 data dirs
>> 10 data dirs
>> 1
>> 14.3 k
>> 2
>> 21.9 k
>> 4
>> 35.8 k
>> 8
>> 24.8 k
>> 43.8 k
>> 72.5 k
>> 77 k
>> 78.6 k
>> 76.6 k
>> 10
>> 58 k
>> 12
>> 49.3 k
>> 49 k
>> Was looking for sweet spot in perf. So did not take measurements for
>>all data points on grid. Only too for the ones that made sense. For
>>example: when perf dropped by adding more sinks, did not take more
>>measurements for those rows.
>> 2. HDFS Sink:
>> Channel: Memory
>> # of HDFS
>> Sinks
>> Snappy
>> BatchSz:1.2mill
>> Snappy
>> BatchSz:1.4mill
>> Sequence File
>> BatchSz:1.2mill
>> 1
>> 34.3 k
>> 33 k
>> 33 k
>> 2
>> 71 k
>> 75 k
>> 69 k
>> 4
>> 141 k
>> 145 k
>> 141 k
>> 8
>> 271 k
>> 273 k
>> 251 k
>> 12
>> 382 k
>> 380 k
>> 370 k
>> 16
>> 478 k
>> 538 k
>> 486 k
>> Some simple observations :
>> * increasing number of dataDirs helps FC perf even on single disk
>>systems
>> * Increasing number of sinks helps
>> * Max throughput observed was about 538k events/sec for HDFS sink
>>which is approx 240MB/s

[jira] [Created] (FLUME-2663) Address Build warnings of duplicate dependencies listed

[Roshan Naik (JIRA)]

Roshan Naik created FLUME-2663:
--

 Summary: Address Build warnings of duplicate dependencies listed
 Key: FLUME-2663
 URL: https://issues.apache.org/jira/browse/FLUME-2663
 Project: Flume
  Issue Type: Bug
  Components: Build
Affects Versions: v1.5.1
Reporter: Roshan Naik
Assignee: Roshan Naik






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2662) Upgrade to Commons-IO 2.4

[Roshan Naik (JIRA)]

Roshan Naik created FLUME-2662:
--

 Summary: Upgrade to Commons-IO 2.4
 Key: FLUME-2662
 URL: https://issues.apache.org/jira/browse/FLUME-2662
 Project: Flume
  Issue Type: Bug
  Components: Build
Affects Versions: v1.5.1
Reporter: Roshan Naik
Assignee: Roshan Naik


Hadoop 2.7 is now switching to apache-commons-io v2.4. Hbase 1.0 is also using 
commons-io v2.4.
Flume is currently at 2.1.
Flume runs into issues like this when tests run against them:

testSequenceFile(org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem)  Time 
elapsed: 77 sec  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:854)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:69)
at 
org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem.testSequenceFile(TestUseRawLocalFileSystem.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

I am planning to submit a patch to upgrade commons-io to 2.4.  Just wanted to 
be cautious and check if we have witnessed has been any issue in the past when 
upgrading apache commons libraries.



Based on what I see here:
http://commons.apache.org/proper/commons-io/upgradeto2_4.html  and

http://commons.apache.org/proper/commons-io/upgradeto2_2.html


Commons-IO 2.4 is binary compat with 2.2 which is in turn binary compat
with 2.1.

There is what they call a "rare" case of source incompat as described in
https://issues.apache.org/jira/browse/IO-318

Doesnt look like we are affected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Upgrading Apache commons-io

[Roshan Naik]

Based on what I see here:
http://commons.apache.org/proper/commons-io/upgradeto2_4.html  and

http://commons.apache.org/proper/commons-io/upgradeto2_2.html


Commons-IO 2.4 is binary compat with 2.2 which is in turn binary compat
with 2.1.

There is what they call a "rare" case of source incompat as described in
https://issues.apache.org/jira/browse/IO-318


Don¹t think we are affected.


-roshan



On 4/8/15 2:03 PM, "Hari Shreedharan"  wrote:

>Is there a backward compat issue?
>
>
>
>
>Thanks, Hari
>
>On Wed, Apr 8, 2015 at 1:59 PM, Roshan Naik 
>wrote:
>
>> Hadoop 2.7 is now switching to apache-commons-io v2.4. Hbase 1.0 is
>>also using commons-io v2.4.
>> Flume is currently at 2.1.
>> Flume runs into issues like this when tests run against them:
>> testSequenceFile(org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem)
>>Time elapsed: 77 sec  <<< ERROR!
>> java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
>> at 
>>org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:854)
>> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
>> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
>> at 
>>org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98
>>)
>> at 
>>org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78
>>)
>> at 
>>org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:69
>>)
>> at 
>>org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem.testSequenceFile(Tes
>>tUseRawLocalFileSystem.java:89)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> I am planning to submit a patch to upgrade commons-io to 2.4.  Just
>>wanted to be cautious and check if we have witnessed has been any issue
>>in the past when upgrading apache commons libraries.
>> -roshan

Re: Upgrading Apache commons-io

[Hari Shreedharan]

Is there a backward compat issue?




Thanks, Hari

On Wed, Apr 8, 2015 at 1:59 PM, Roshan Naik 
wrote:

> Hadoop 2.7 is now switching to apache-commons-io v2.4. Hbase 1.0 is also 
> using commons-io v2.4.
> Flume is currently at 2.1.
> Flume runs into issues like this when tests run against them:
> testSequenceFile(org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem)  Time 
> elapsed: 77 sec  <<< ERROR!
> java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
> at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:854)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:69)
> at 
> org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem.testSequenceFile(TestUseRawLocalFileSystem.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> I am planning to submit a patch to upgrade commons-io to 2.4.  Just wanted to 
> be cautious and check if we have witnessed has been any issue in the past 
> when upgrading apache commons libraries.
> -roshan

Upgrading Apache commons-io

[Roshan Naik]

Hadoop 2.7 is now switching to apache-commons-io v2.4. Hbase 1.0 is also using 
commons-io v2.4.
Flume is currently at 2.1.
Flume runs into issues like this when tests run against them:

testSequenceFile(org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem)  Time 
elapsed: 77 sec  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:854)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:69)
at 
org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem.testSequenceFile(TestUseRawLocalFileSystem.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

I am planning to submit a patch to upgrade commons-io to 2.4.  Just wanted to 
be cautious and check if we have witnessed has been any issue in the past when 
upgrading apache commons libraries.

-roshan

[jira] [Commented] (FLUME-2595) Add option to checkpoint on file channel shutdown

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485958#comment-14485958
 ] 

Roshan Naik commented on FLUME-2595:


Ok. Will try to look into it today.

> Add option to checkpoint on file channel shutdown
> -
>
> Key: FLUME-2595
> URL: https://issues.apache.org/jira/browse/FLUME-2595
> Project: Flume
>  Issue Type: Bug
>  Components: File Channel
>Affects Versions: v1.5.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: channel
> Attachments: FLUME-2595.patch, FLUME-2595.v2.patch
>
>
> Restarting a File channel based agent can take very long, a few hours to 
> evens days in some cases. The reason being the slow replay performance.
> By having the file channel checkpoint on shutdown, this long replay can 
> avoided as there will be nothing to replay on startup.
> A small hit (a few seconds perhaps) will be observed on shutdown to perform 
> the checkpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2375) HDFS sink's fail to recover from datanode unavailability

[Jeff Field (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485561#comment-14485561
 ] 

Jeff Field commented on FLUME-2375:
---

We're also seeing this problem regularly and can reproduce it at-will, and have 
logs available/are willing to test fixes.

> HDFS sink's fail to recover from datanode unavailability
> 
>
> Key: FLUME-2375
> URL: https://issues.apache.org/jira/browse/FLUME-2375
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.4.0
>Reporter: David Stendardi
>  Labels: hdfs, hdfssink
>
> Hello !
> We are running flume-ng with version cdh-4.5-1.4. When a datanode used by 
> flume-ng goes done, we get the following exceptions :  
> {code}
> 30 Apr 2014 01:10:38,130 ERROR 
> [SinkRunner-PollingRunner-DefaultSinkProcessor] 
> (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:96)  - 
> Unexpected error while checking replication factor
> java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:162)
> at 
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:82)
> at 
> org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:452)
> at 
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:387)
> at 
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> These exceptions are logged but not rethrown, and the 
> AbstractHdfsSink::isUnderReplicated still returns false so the writer 
> continue to try writing on the node.
> Here is how we configured our sink : 
> {code}
> collector.sinks.hdfs.channel = hdfs
> collector.sinks.hdfs.type = hdfs
> collector.sinks.hdfs.hdfs.path = 
> /flume-ng/%{env}/%{avro.fqn}/from_year=%Y/from_date=%Y-%m-%d
> collector.sinks.hdfs.hdfs.filePrefix = <%= @hostname %>-%H-%{avro.fp}
> collector.sinks.hdfs.hdfs.fileSuffix = .avro
> collector.sinks.hdfs.hdfs.rollInterval = 3605
> collector.sinks.hdfs.hdfs.rollSize = 0
> collector.sinks.hdfs.hdfs.rollCount = 0
> collector.sinks.hdfs.hdfs.batchSize = 1000
> collector.sinks.hdfs.hdfs.txnEventMax = 1000
> collector.sinks.hdfs.hdfs.callTimeout = 2
> collector.sinks.hdfs.hdfs.fileType = DataStream
> collector.sinks.hdfs.serializer = 
> com.viadeo.event.flume.serializer.AvroEventSerializer$Builder
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)