Re: Flume startup takes ~ hour

2013-09-24 Thread Anat Rozenzon
After some deeper dive, it seems that the problem is with HashMap usage in EventQueueBackingStoreFile. Almost every time I run jstack the JVM is inside EventQueueBackingStoreFile.get() doing either HashMap.containsKey() or Integer.valueOf(). This is because of overwriteMap is defined as regular

Re: Flume startup takes ~ hour

2013-09-24 Thread Anat Rozenzon
For example this stack trace: lifecycleSupervisor-1-2 prio=10 tid=0x7f89141d8800 nid=0x5ac8 runnable [0x7f89501ad000] java.lang.Thread.State: RUNNABLE at java.lang.Integer.valueOf(Integer.java:642) at

Re: Flume startup takes ~ hour

2013-09-24 Thread Hari Shreedharan
That is actually a symptom of the real problem. The real problem is that the remove method ends up hitting the main checkpoint data structure and causes too many ops on the hash map. The real fix is in the patch I mentioned which reduce the number of ops tremendously. Thanks, Hari On

[ANNOUNCE] New Flume Committer - Roshan Naik

2013-09-24 Thread Hari Shreedharan
On behalf of the Apache Flume PMC, I am excited to welcome Roshan Naik as a committer on the Apache Flume project. Roshan has actively contributed several patches to the Flume project, including bug fixes, Windows support and new features. Congratulations and Welcome, Roshan! Cheers, Hari

Re: [ANNOUNCE] New Flume Committer - Wolfgang Hoschek

2013-09-24 Thread Jarek Jarcec Cecho
Congratulations Wolfgang, well done! Jarcec On Tue, Sep 24, 2013 at 03:39:12PM -0700, Hari Shreedharan wrote: On behalf of the Apache Flume PMC, I am excited to welcome Wolfgang Hoschek as a committer on the Apache Flume project. Wolfgang contributed a new sink with the ability to do

Re: [ANNOUNCE] New Flume Committer - Roshan Naik

2013-09-24 Thread Jarek Jarcec Cecho
Congratulations Roshan, well done! Jarcec On Tue, Sep 24, 2013 at 03:39:13PM -0700, Hari Shreedharan wrote: On behalf of the Apache Flume PMC, I am excited to welcome Roshan Naik as a committer on the Apache Flume project. Roshan has actively contributed several patches to the Flume project,

Re: [ANNOUNCE] New Flume Committer - Wolfgang Hoschek

2013-09-24 Thread Mike Percy
Congrats Wolfgang, and welcome! Mike On Tue, Sep 24, 2013 at 3:46 PM, Jarek Jarcec Cecho jar...@apache.orgwrote: Congratulations Wolfgang, well done! Jarcec On Tue, Sep 24, 2013 at 03:39:12PM -0700, Hari Shreedharan wrote: On behalf of the Apache Flume PMC, I am excited to welcome

Re: [ANNOUNCE] New Flume Committer - Roshan Naik

2013-09-24 Thread Mike Percy
Congrats Roshan, welcome! Mike On Tue, Sep 24, 2013 at 3:47 PM, Jarek Jarcec Cecho jar...@apache.orgwrote: Congratulations Roshan, well done! Jarcec On Tue, Sep 24, 2013 at 03:39:13PM -0700, Hari Shreedharan wrote: On behalf of the Apache Flume PMC, I am excited to welcome Roshan Naik

Re: [ANNOUNCE] New Flume Committer - Roshan Naik

2013-09-24 Thread Arvind Prabhakar
Congratulations Roshan! Regards, Arvind Prabhakar On Tue, Sep 24, 2013 at 4:05 PM, Mike Percy mpe...@apache.org wrote: Congrats Roshan, welcome! Mike On Tue, Sep 24, 2013 at 3:47 PM, Jarek Jarcec Cecho jar...@apache.orgwrote: Congratulations Roshan, well done! Jarcec On Tue, Sep

Re: [ANNOUNCE] New Flume Committer - Wolfgang Hoschek

2013-09-24 Thread Arvind Prabhakar
Congratulations Wolfgang! Regards, Arvind Prabhakar On Tue, Sep 24, 2013 at 4:05 PM, Mike Percy mpe...@apache.org wrote: Congrats Wolfgang, and welcome! Mike On Tue, Sep 24, 2013 at 3:46 PM, Jarek Jarcec Cecho jar...@apache.orgwrote: Congratulations Wolfgang, well done! Jarcec On

Re: [ANNOUNCE] New Flume Committer - Wolfgang Hoschek

2013-09-24 Thread Mubarak Seyed
Congratulations Wolfgang! Thanks. -Mubarak On Sep 24, 2013, at 10:52 PM, Arvind Prabhakar arv...@apache.org wrote: Congratulations Wolfgang! Regards, Arvind Prabhakar On Tue, Sep 24, 2013 at 4:05 PM, Mike Percy mpe...@apache.org wrote: Congrats Wolfgang, and welcome! Mike