Using CodaHale metrics for monitoring

[Ashish]

Hi,

Have hacked a bit into our existing instrumentation package and piggy
backed cohahale metrics package. Here is one sample for Spooled
Directory source (with instrumentation only for Source and Channel ),
using console reporter

-- Gauges --

org.apache.flume.instrumentation.ChannelCounter.channel.current.size

 value = 200

org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage

 value = 2.0

org.apache.flume.instrumentation.SourceCounter.src.open-connection.count

 value = 0


-- Counters 

org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt

 count = 1138800

org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success

 count = 1138800

org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt

 count = 1138601

org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success

 count = 1138600

org.apache.flume.instrumentation.SourceCounter.src.events.accepted

 count = 1138800

org.apache.flume.instrumentation.SourceCounter.src.events.received

 count = 1138800

src.append-batch.accepted

 count = 11388

src.append-batch.received

 count = 11388

src.append.accepted

 count = 0

src.append.received

 count = 0


-- Meters --

eventAcceptedMeter

 count = 1138800

 mean rate = 106478.56 events/second

 1-minute rate = 93420.18 events/second

 5-minute rate = 91389.24 events/second

15-minute rate = 91037.40 events/second

eventReceivedMeter

 count = 1138800

 mean rate = 106462.14 events/second

 1-minute rate = 93420.18 events/second

 5-minute rate = 91389.24 events/second

15-minute rate = 91037.40 events/second


If there is interest in the community, can raise a jira and continue
to work on it.


-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

[jira] [Commented] (FLUME-2502) Spool source's directory listing is inefficient

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203611#comment-14203611
 ] 

Hudson commented on FLUME-2502:
---

SUCCESS: Integrated in Flume-trunk-hbase-98 #47 (See 
[https://builds.apache.org/job/Flume-trunk-hbase-98/47/])
FLUME-2505: Test added in FLUME-2502 is flaky (jarcec: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=76ddc82fb829995f2c88976d4153de132edf0a20)
* 
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java


> Spool source's directory listing is inefficient
> ---
>
> Key: FLUME-2502
> URL: https://issues.apache.org/jira/browse/FLUME-2502
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.0
>Reporter: Prateek Rungta
>Assignee: Prateek Rungta
> Attachments: FLUME-2502-0.patch, FLUME-2502-1.patch, 
> FLUME-2502-2.patch, FLUME-2502-3.patch, FLUME-2502-final.patch
>
>
> As mentioned in 
> [FLUME-2309|https://issues.apache.org/jira/browse/FLUME-2309], the directory 
> listing can it self become the bottleneck when accessing directories with a 
> large number of files (>1M). The fix in that JIRA added in the ability to 
> specify `RANDOM` as a Consume-Order to avoid sorting large lists.
> The slowness of the directory listing is still un-addressed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : Flume-trunk-hbase-98 #47

[Apache Jenkins Server]

See 

[jira] [Commented] (FLUME-2505) Test added in FLUME-2502 is flaky

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203610#comment-14203610
 ] 

Hudson commented on FLUME-2505:
---

SUCCESS: Integrated in Flume-trunk-hbase-98 #47 (See 
[https://builds.apache.org/job/Flume-trunk-hbase-98/47/])
FLUME-2505: Test added in FLUME-2502 is flaky (jarcec: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=76ddc82fb829995f2c88976d4153de132edf0a20)
* 
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java


> Test added in FLUME-2502 is flaky
> -
>
> Key: FLUME-2505
> URL: https://issues.apache.org/jira/browse/FLUME-2505
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2505.patch, FLUME-2505.patch
>
>
> I added a test to Prateek's patch - which is flaky on Jenkins (not locally) - 
> probably due to slower machines. I think we should make the test a bit more 
> tolerant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2502) Spool source's directory listing is inefficient

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203609#comment-14203609
 ] 

Hudson commented on FLUME-2502:
---

SUCCESS: Integrated in flume-trunk #690 (See 
[https://builds.apache.org/job/flume-trunk/690/])
FLUME-2505: Test added in FLUME-2502 is flaky (jarcec: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=76ddc82fb829995f2c88976d4153de132edf0a20)
* 
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java


> Spool source's directory listing is inefficient
> ---
>
> Key: FLUME-2502
> URL: https://issues.apache.org/jira/browse/FLUME-2502
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.0
>Reporter: Prateek Rungta
>Assignee: Prateek Rungta
> Attachments: FLUME-2502-0.patch, FLUME-2502-1.patch, 
> FLUME-2502-2.patch, FLUME-2502-3.patch, FLUME-2502-final.patch
>
>
> As mentioned in 
> [FLUME-2309|https://issues.apache.org/jira/browse/FLUME-2309], the directory 
> listing can it self become the bottleneck when accessing directories with a 
> large number of files (>1M). The fix in that JIRA added in the ability to 
> specify `RANDOM` as a Consume-Order to avoid sorting large lists.
> The slowness of the directory listing is still un-addressed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2505) Test added in FLUME-2502 is flaky

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203608#comment-14203608
 ] 

Hudson commented on FLUME-2505:
---

SUCCESS: Integrated in flume-trunk #690 (See 
[https://builds.apache.org/job/flume-trunk/690/])
FLUME-2505: Test added in FLUME-2502 is flaky (jarcec: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=76ddc82fb829995f2c88976d4153de132edf0a20)
* 
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java


> Test added in FLUME-2502 is flaky
> -
>
> Key: FLUME-2505
> URL: https://issues.apache.org/jira/browse/FLUME-2505
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2505.patch, FLUME-2505.patch
>
>
> I added a test to Prateek's patch - which is flaky on Jenkins (not locally) - 
> probably due to slower machines. I think we should make the test a bit more 
> tolerant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2502) Spool source's directory listing is inefficient

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203576#comment-14203576
 ] 

ASF subversion and git services commented on FLUME-2502:


Commit 76ddc82fb829995f2c88976d4153de132edf0a20 in flume's branch 
refs/heads/trunk from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=76ddc82 ]

FLUME-2505: Test added in FLUME-2502 is flaky

(Hari Shreedharan via Jarek Jarcec Cecho)


> Spool source's directory listing is inefficient
> ---
>
> Key: FLUME-2502
> URL: https://issues.apache.org/jira/browse/FLUME-2502
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.0
>Reporter: Prateek Rungta
>Assignee: Prateek Rungta
> Attachments: FLUME-2502-0.patch, FLUME-2502-1.patch, 
> FLUME-2502-2.patch, FLUME-2502-3.patch, FLUME-2502-final.patch
>
>
> As mentioned in 
> [FLUME-2309|https://issues.apache.org/jira/browse/FLUME-2309], the directory 
> listing can it self become the bottleneck when accessing directories with a 
> large number of files (>1M). The fix in that JIRA added in the ability to 
> specify `RANDOM` as a Consume-Order to avoid sorting large lists.
> The slowness of the directory listing is still un-addressed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2505) Test added in FLUME-2502 is flaky

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203577#comment-14203577
 ] 

ASF subversion and git services commented on FLUME-2505:


Commit 4976f587a76c4d9b7acd291b73cd7a60427be130 in flume's branch 
refs/heads/flume-1.6 from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4976f58 ]

FLUME-2505: Test added in FLUME-2502 is flaky

(Hari Shreedharan via Jarek Jarcec Cecho)


> Test added in FLUME-2502 is flaky
> -
>
> Key: FLUME-2505
> URL: https://issues.apache.org/jira/browse/FLUME-2505
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2505.patch, FLUME-2505.patch
>
>
> I added a test to Prateek's patch - which is flaky on Jenkins (not locally) - 
> probably due to slower machines. I think we should make the test a bit more 
> tolerant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2502) Spool source's directory listing is inefficient

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203578#comment-14203578
 ] 

ASF subversion and git services commented on FLUME-2502:


Commit 4976f587a76c4d9b7acd291b73cd7a60427be130 in flume's branch 
refs/heads/flume-1.6 from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4976f58 ]

FLUME-2505: Test added in FLUME-2502 is flaky

(Hari Shreedharan via Jarek Jarcec Cecho)


> Spool source's directory listing is inefficient
> ---
>
> Key: FLUME-2502
> URL: https://issues.apache.org/jira/browse/FLUME-2502
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.0
>Reporter: Prateek Rungta
>Assignee: Prateek Rungta
> Attachments: FLUME-2502-0.patch, FLUME-2502-1.patch, 
> FLUME-2502-2.patch, FLUME-2502-3.patch, FLUME-2502-final.patch
>
>
> As mentioned in 
> [FLUME-2309|https://issues.apache.org/jira/browse/FLUME-2309], the directory 
> listing can it self become the bottleneck when accessing directories with a 
> large number of files (>1M). The fix in that JIRA added in the ability to 
> specify `RANDOM` as a Consume-Order to avoid sorting large lists.
> The slowness of the directory listing is still un-addressed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2505) Test added in FLUME-2502 is flaky

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203575#comment-14203575
 ] 

ASF subversion and git services commented on FLUME-2505:


Commit 76ddc82fb829995f2c88976d4153de132edf0a20 in flume's branch 
refs/heads/trunk from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=76ddc82 ]

FLUME-2505: Test added in FLUME-2502 is flaky

(Hari Shreedharan via Jarek Jarcec Cecho)


> Test added in FLUME-2502 is flaky
> -
>
> Key: FLUME-2505
> URL: https://issues.apache.org/jira/browse/FLUME-2505
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2505.patch, FLUME-2505.patch
>
>
> I added a test to Prateek's patch - which is flaky on Jenkins (not locally) - 
> probably due to slower machines. I think we should make the test a bit more 
> tolerant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2505) Test added in FLUME-2502 is flaky

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203574#comment-14203574
 ] 

Jarek Jarcec Cecho commented on FLUME-2505:
---

+1

> Test added in FLUME-2502 is flaky
> -
>
> Key: FLUME-2505
> URL: https://issues.apache.org/jira/browse/FLUME-2505
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2505.patch, FLUME-2505.patch
>
>
> I added a test to Prateek's patch - which is flaky on Jenkins (not locally) - 
> probably due to slower machines. I think we should make the test a bit more 
> tolerant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-1521) Document the StressSource

[Ashish Paliwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203416#comment-14203416
 ] 

Ashish Paliwal commented on FLUME-1521:
---

[~hshreedharan] can you please review it? minor one, might need rebasing

> Document the StressSource
> -
>
> Key: FLUME-1521
> URL: https://issues.apache.org/jira/browse/FLUME-1521
> Project: Flume
>  Issue Type: Improvement
>  Components: Docs
>Affects Versions: v1.2.0, v1.3.0, v1.4.0
>Reporter: Denny Ye
>Assignee: Ashish Paliwal
>Priority: Minor
> Attachments: FLUME-1521-1.patch, FLUME-1521.patch
>
>
> StressSource is absent at UserGuide, we should fix it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Usage of CounterGroup class?

[Ashish]

Hi,

Can someone help me understand the usage of CounterGroup class. Can
see its usage for collecting metrics, but how is it reported (via
logging only?) The usage is also not consistent across Sources/Sinks,
like Avro source doesn't use, neither does HDFS/Kafka sinks.

The class has all the methods as synchronized. The only time it is
needed is while adding the counter to the map, rest synchronisation is
handled by AtomicLong class. We can register the counters while
initializing the class and get rid of synchronized clause from class.
It's not a big issue, AFAIK JVM would anyways be optimising it during
runtime.

Thoughts?

-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal