[jira] [Created] (FLUME-2550) Revamp Metrics reporting to use CodaHale Metrics package

[Ashish Paliwal (JIRA)]

Ashish Paliwal created FLUME-2550:
-

 Summary: Revamp Metrics reporting to use CodaHale Metrics package
 Key: FLUME-2550
 URL: https://issues.apache.org/jira/browse/FLUME-2550
 Project: Flume
  Issue Type: Improvement
  Components: Configuration
Affects Versions: v1.5.0
Reporter: Ashish Paliwal
Assignee: Ashish Paliwal


Use CodaHale Metrics package to revamp current Metrics reporting system. This 
shall help provide lot more information around our existing Metrics



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Using CodaHale metrics for monitoring

[Hari Shreedharan]

Makes sense.


Thanks,
Hari

On Tue, Nov 11, 2014 at 9:51 PM, Ashish  wrote:

> That's what I had in mind, just need to figure out how to do it with
> CodaHale. Want to avoid CodaHale MetricRegistry class floating around
> in code. 1st step would be to migrate to CodaHale, I am yet to figure
> out how managing the Reporter part. Let me try to put a patch
> together.
> On Wed, Nov 12, 2014 at 9:29 AM, Hari Shreedharan
>  wrote:
>> Currently the metrics exposed via Sources/sinks/channels are hardcoded. We 
>> could add a method to add custom metrics (I think it is already there in 
>> MonitoredCounterGroup, just not in the *Counter classes).
>>
>>
>> Thanks,
>> Hari
>>
>> On Tue, Nov 11, 2014 at 6:34 PM, Ashish  wrote:
>>
>>> I was just playing around so piggy-backed it. The changes have to be
>>> within MonitoredCounterGroup and its sub-classes, as its used across
>>> the implementations. Let me raise a JIRA for this.
>>> One thing that I would like to have in future is to allow
>>> Sinks/Channels/Sources to be able to have custom counters/Meters and
>>> be available to have those Metrics available via the framework. I
>>> think CounterGroup was created for the same purpose.
>>> Let me create a JIRA for this and work on it.
>>> thanks
>>> ashish
>>> On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
>>>  wrote:
 I think it is best to replace the current system rather than having 
 multiple metrics systems. We should implement it such that it is 
 transparent to user code, and the changes only go into 
 MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).


 Thanks,
 Hari

 On Mon, Nov 10, 2014 at 5:36 PM, Ashish  wrote:

> Codahale does have lot more features like the rate/sec one, which is
> most needed metrics. It provides a lot of reporters out of box
> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
> enable it, rather than writing custom components.
> As of now we have to hide it within MonitoredCounterGroup and its
> sub-classes, that's what I did for running it. It was running parallel
> to existing system. Let me work more on it and see if I can simplify
> the same.
> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>  wrote:
>> Is it easier to use than the current one and/or does it give better 
>> performance? You’d need to support the current metrics API 
>> (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>
>>
>> Thanks,
>> Hari
>>
>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish  wrote:
>>
>>> Hi,
>>> Have hacked a bit into our existing instrumentation package and piggy
>>> backed cohahale metrics package. Here is one sample for Spooled
>>> Directory source (with instrumentation only for Source and Channel ),
>>> using console reporter
>>> -- Gauges 
>>> --
>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>>  value = 200
>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>>  value = 2.0
>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>>  value = 0
>>> -- Counters 
>>> 
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>>  count = 1138800
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>>  count = 1138800
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>>  count = 1138601
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>>  count = 1138600
>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>>  count = 1138800
>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>>  count = 1138800
>>> src.append-batch.accepted
>>>  count = 11388
>>> src.append-batch.received
>>>  count = 11388
>>> src.append.accepted
>>>  count = 0
>>> src.append.received
>>>  count = 0
>>> -- Meters 
>>> --
>>> eventAcceptedMeter
>>>  count = 1138800
>>>  mean rate = 106478.56 events/second
>>>  1-minute rate = 93420.18 events/second
>>>  5-minute rate = 91389.24 events/second
>>> 15-minute rate = 91037.40 events/second
>>> eventReceivedMeter
>>>  count = 1138800
>>>  mean rate = 106462.14 events/second
>>>  1-minute rate = 93420.18 events/second
>>>  5-minute rate = 91389.24 events/second
>>> 15-minute

Re: Using CodaHale metrics for monitoring

[Ashish]

That's what I had in mind, just need to figure out how to do it with
CodaHale. Want to avoid CodaHale MetricRegistry class floating around
in code. 1st step would be to migrate to CodaHale, I am yet to figure
out how managing the Reporter part. Let me try to put a patch
together.

On Wed, Nov 12, 2014 at 9:29 AM, Hari Shreedharan
 wrote:
> Currently the metrics exposed via Sources/sinks/channels are hardcoded. We 
> could add a method to add custom metrics (I think it is already there in 
> MonitoredCounterGroup, just not in the *Counter classes).
>
>
> Thanks,
> Hari
>
> On Tue, Nov 11, 2014 at 6:34 PM, Ashish  wrote:
>
>> I was just playing around so piggy-backed it. The changes have to be
>> within MonitoredCounterGroup and its sub-classes, as its used across
>> the implementations. Let me raise a JIRA for this.
>> One thing that I would like to have in future is to allow
>> Sinks/Channels/Sources to be able to have custom counters/Meters and
>> be available to have those Metrics available via the framework. I
>> think CounterGroup was created for the same purpose.
>> Let me create a JIRA for this and work on it.
>> thanks
>> ashish
>> On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
>>  wrote:
>>> I think it is best to replace the current system rather than having 
>>> multiple metrics systems. We should implement it such that it is 
>>> transparent to user code, and the changes only go into 
>>> MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Mon, Nov 10, 2014 at 5:36 PM, Ashish  wrote:
>>>
 Codahale does have lot more features like the rate/sec one, which is
 most needed metrics. It provides a lot of reporters out of box
 (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
 enable it, rather than writing custom components.
 As of now we have to hide it within MonitoredCounterGroup and its
 sub-classes, that's what I did for running it. It was running parallel
 to existing system. Let me work more on it and see if I can simplify
 the same.
 On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
  wrote:
> Is it easier to use than the current one and/or does it give better 
> performance? You’d need to support the current metrics API 
> (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>
>
> Thanks,
> Hari
>
> On Sat, Nov 8, 2014 at 8:21 PM, Ashish  wrote:
>
>> Hi,
>> Have hacked a bit into our existing instrumentation package and piggy
>> backed cohahale metrics package. Here is one sample for Spooled
>> Directory source (with instrumentation only for Source and Channel ),
>> using console reporter
>> -- Gauges 
>> --
>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>  value = 200
>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>  value = 2.0
>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>  value = 0
>> -- Counters 
>> 
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>  count = 1138800
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>  count = 1138800
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>  count = 1138601
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>  count = 1138600
>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>  count = 1138800
>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>  count = 1138800
>> src.append-batch.accepted
>>  count = 11388
>> src.append-batch.received
>>  count = 11388
>> src.append.accepted
>>  count = 0
>> src.append.received
>>  count = 0
>> -- Meters 
>> --
>> eventAcceptedMeter
>>  count = 1138800
>>  mean rate = 106478.56 events/second
>>  1-minute rate = 93420.18 events/second
>>  5-minute rate = 91389.24 events/second
>> 15-minute rate = 91037.40 events/second
>> eventReceivedMeter
>>  count = 1138800
>>  mean rate = 106462.14 events/second
>>  1-minute rate = 93420.18 events/second
>>  5-minute rate = 91389.24 events/second
>> 15-minute rate = 91037.40 events/second
>> If there is interest in the community, can raise a jira and continue
>> to work on it.
>> --
>> thanks
>> ashish
>> Blog: http://www.as

[jira] [Comment Edited] (FLUME-2307) Remove Log writetimeout

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207651#comment-14207651
 ] 

Hari Shreedharan edited comment on FLUME-2307 at 11/12/14 4:10 AM:
---

>From the logs you have here, I can't really see what the issue is. What is the 
>exact version of Flume you are running? 

flume-ng version can give you that info.


was (Author: hshreedharan):
>From the logs you have here, I can't really see what the issue is. What is the 
>exact version of Flume you are running? 

`flume-ng version` can give you that info.

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> 

[jira] [Commented] (FLUME-2307) Remove Log writetimeout

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207651#comment-14207651
 ] 

Hari Shreedharan commented on FLUME-2307:
-

>From the logs you have here, I can't really see what the issue is. What is the 
>exact version of Flume you are running? 

`flume-ng version` can give you that info.

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at 
> org.jboss.netty.handler.

[jira] [Commented] (FLUME-2307) Remove Log writetimeout

[Nina Safonova (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207648#comment-14207648
 ] 

Nina Safonova commented on FLUME-2307:
--

I didn't delete any files before I start to experience this issue. After I ran 
out of disk space I tried to restart the agent. When it didn't help to clean 
old files I stopped the agent and deleted all the files manually.

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder

[jira] [Commented] (FLUME-2307) Remove Log writetimeout

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207645#comment-14207645
 ] 

Hari Shreedharan commented on FLUME-2307:
-

>From your logs, I don't see the actual cause of this issue. Did you also 
>delete the inflight* files too?

Deleting the data files and keeping any of the checkpoint files can cause 
unpredictable issues. 

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)

Re: Using CodaHale metrics for monitoring

[Hari Shreedharan]

Currently the metrics exposed via Sources/sinks/channels are hardcoded. We 
could add a method to add custom metrics (I think it is already there in 
MonitoredCounterGroup, just not in the *Counter classes).


Thanks,
Hari

On Tue, Nov 11, 2014 at 6:34 PM, Ashish  wrote:

> I was just playing around so piggy-backed it. The changes have to be
> within MonitoredCounterGroup and its sub-classes, as its used across
> the implementations. Let me raise a JIRA for this.
> One thing that I would like to have in future is to allow
> Sinks/Channels/Sources to be able to have custom counters/Meters and
> be available to have those Metrics available via the framework. I
> think CounterGroup was created for the same purpose.
> Let me create a JIRA for this and work on it.
> thanks
> ashish
> On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
>  wrote:
>> I think it is best to replace the current system rather than having multiple 
>> metrics systems. We should implement it such that it is transparent to user 
>> code, and the changes only go into MonitoredCounterGroup (the CounterGroup 
>> class is obsolete - dont bother).
>>
>>
>> Thanks,
>> Hari
>>
>> On Mon, Nov 10, 2014 at 5:36 PM, Ashish  wrote:
>>
>>> Codahale does have lot more features like the rate/sec one, which is
>>> most needed metrics. It provides a lot of reporters out of box
>>> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
>>> enable it, rather than writing custom components.
>>> As of now we have to hide it within MonitoredCounterGroup and its
>>> sub-classes, that's what I did for running it. It was running parallel
>>> to existing system. Let me work more on it and see if I can simplify
>>> the same.
>>> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>>>  wrote:
 Is it easier to use than the current one and/or does it give better 
 performance? You’d need to support the current metrics API 
 (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).


 Thanks,
 Hari

 On Sat, Nov 8, 2014 at 8:21 PM, Ashish  wrote:

> Hi,
> Have hacked a bit into our existing instrumentation package and piggy
> backed cohahale metrics package. Here is one sample for Spooled
> Directory source (with instrumentation only for Source and Channel ),
> using console reporter
> -- Gauges 
> --
> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>  value = 200
> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>  value = 2.0
> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>  value = 0
> -- Counters 
> 
> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>  count = 1138800
> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>  count = 1138800
> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>  count = 1138601
> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>  count = 1138600
> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>  count = 1138800
> org.apache.flume.instrumentation.SourceCounter.src.events.received
>  count = 1138800
> src.append-batch.accepted
>  count = 11388
> src.append-batch.received
>  count = 11388
> src.append.accepted
>  count = 0
> src.append.received
>  count = 0
> -- Meters 
> --
> eventAcceptedMeter
>  count = 1138800
>  mean rate = 106478.56 events/second
>  1-minute rate = 93420.18 events/second
>  5-minute rate = 91389.24 events/second
> 15-minute rate = 91037.40 events/second
> eventReceivedMeter
>  count = 1138800
>  mean rate = 106462.14 events/second
>  1-minute rate = 93420.18 events/second
>  5-minute rate = 91389.24 events/second
> 15-minute rate = 91037.40 events/second
> If there is interest in the community, can raise a jira and continue
> to work on it.
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>> --
>>> thanks
>>> ashish
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
> -- 
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal

[jira] [Commented] (FLUME-2307) Remove Log writetimeout

[Nina Safonova (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207643#comment-14207643
 ] 

Nina Safonova commented on FLUME-2307:
--

I didn't try to delete checkpoint files.
To keep processing data I deleted all the files (data and checkpoints) after I 
stopped the agent.

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageR

[jira] [Commented] (FLUME-2307) Remove Log writetimeout

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207640#comment-14207640
 ] 

Hari Shreedharan commented on FLUME-2307:
-

This does not seem like it is due to log timeout. Can you delete all of the 
checkpoint files (all files) and force a full replay?

Did you delete the files before or after stopping the agent?

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at 

[jira] [Commented] (FLUME-2307) Remove Log writetimeout

[Nina Safonova (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207623#comment-14207623
 ] 

Nina Safonova commented on FLUME-2307:
--

OS is CentOS 6.5, flume version is 1.5 as I mentioned above, all the log I also 
posted above.
I waited log enough and no old files were deleted. This doesn't work that way 
all the time, from the start to some random moment it's working as expected and 
cleans old files, but at some point it just stops to do this and at some later 
point disk ran out of space.

> Remove Log writetimeout
> ---
>
> Key: FLUME-2307
> URL: https://issues.apache.org/jira/browse/FLUME-2307
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.4.0
>Reporter: Steve Zesch
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2307-1.patch, FLUME-2307.patch
>
>
> I've observed Flume failing to clean up old log data in FileChannels. The 
> amount of old log data can range anywhere from tens to hundreds of GB. I was 
> able to confirm that the channels were in fact empty. This behavior always 
> occurs after lock timeouts when attempting to put, take, rollback, or commit 
> to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old 
> files. I was able to confirm that the Log's writeCheckpoint method was still 
> being called and successfully obtaining a lock from tryLockExclusive(), but I 
> was not able to confirm removeOldLogs being called. The application log did 
> not include "Removing old file: log-xyz" for the old files which the Log 
> class would output if they were correctly being removed. I suspect the lock 
> timeouts were due to high I/O load at the time.
> Some stack traces:
> {code}
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at 
> dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
> at 
> dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
> at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> org.apache.flume.ChannelException: Failed to obtain lock for writing to the 
> log. Try increasing the log write timeout value. [channel=fileChannel]
> at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
> at 
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
> at 
> dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
> at org.apache.avro.ipc.Responder.respond(Responder.java:151)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at 
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.

Re: Usage of CounterGroup class?

[Ashish]

CounterGroup is obsolete (got it on another thread).

Just closing this Thread to avoid Zombies :)

On Sat, Nov 8, 2014 at 6:18 PM, Ashish  wrote:
> Hi,
>
> Can someone help me understand the usage of CounterGroup class. Can
> see its usage for collecting metrics, but how is it reported (via
> logging only?) The usage is also not consistent across Sources/Sinks,
> like Avro source doesn't use, neither does HDFS/Kafka sinks.
>
> The class has all the methods as synchronized. The only time it is
> needed is while adding the counter to the map, rest synchronisation is
> handled by AtomicLong class. We can register the counters while
> initializing the class and get rid of synchronized clause from class.
> It's not a big issue, AFAIK JVM would anyways be optimising it during
> runtime.
>
> Thoughts?
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: Using CodaHale metrics for monitoring

[Ashish]

I was just playing around so piggy-backed it. The changes have to be
within MonitoredCounterGroup and its sub-classes, as its used across
the implementations. Let me raise a JIRA for this.

One thing that I would like to have in future is to allow
Sinks/Channels/Sources to be able to have custom counters/Meters and
be available to have those Metrics available via the framework. I
think CounterGroup was created for the same purpose.

Let me create a JIRA for this and work on it.

thanks
ashish

On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
 wrote:
> I think it is best to replace the current system rather than having multiple 
> metrics systems. We should implement it such that it is transparent to user 
> code, and the changes only go into MonitoredCounterGroup (the CounterGroup 
> class is obsolete - dont bother).
>
>
> Thanks,
> Hari
>
> On Mon, Nov 10, 2014 at 5:36 PM, Ashish  wrote:
>
>> Codahale does have lot more features like the rate/sec one, which is
>> most needed metrics. It provides a lot of reporters out of box
>> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
>> enable it, rather than writing custom components.
>> As of now we have to hide it within MonitoredCounterGroup and its
>> sub-classes, that's what I did for running it. It was running parallel
>> to existing system. Let me work more on it and see if I can simplify
>> the same.
>> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>>  wrote:
>>> Is it easier to use than the current one and/or does it give better 
>>> performance? You’d need to support the current metrics API 
>>> (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish  wrote:
>>>
 Hi,
 Have hacked a bit into our existing instrumentation package and piggy
 backed cohahale metrics package. Here is one sample for Spooled
 Directory source (with instrumentation only for Source and Channel ),
 using console reporter
 -- Gauges 
 --
 org.apache.flume.instrumentation.ChannelCounter.channel.current.size
  value = 200
 org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
  value = 2.0
 org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
  value = 0
 -- Counters 
 
 org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
  count = 1138800
 org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
  count = 1138800
 org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
  count = 1138601
 org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
  count = 1138600
 org.apache.flume.instrumentation.SourceCounter.src.events.accepted
  count = 1138800
 org.apache.flume.instrumentation.SourceCounter.src.events.received
  count = 1138800
 src.append-batch.accepted
  count = 11388
 src.append-batch.received
  count = 11388
 src.append.accepted
  count = 0
 src.append.received
  count = 0
 -- Meters 
 --
 eventAcceptedMeter
  count = 1138800
  mean rate = 106478.56 events/second
  1-minute rate = 93420.18 events/second
  5-minute rate = 91389.24 events/second
 15-minute rate = 91037.40 events/second
 eventReceivedMeter
  count = 1138800
  mean rate = 106462.14 events/second
  1-minute rate = 93420.18 events/second
  5-minute rate = 91389.24 events/second
 15-minute rate = 91037.40 events/second
 If there is interest in the community, can raise a jira and continue
 to work on it.
 --
 thanks
 ashish
 Blog: http://www.ashishpaliwal.com/blog
 My Photo Galleries: http://www.pbase.com/ashishpaliwal
>> --
>> thanks
>> ashish
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: Review Request 18544: Hive Streaming sink

[Roshan Naik]

> On Aug. 21, 2014, 7:24 p.m., Brock Noland wrote:
> > conf/log4j.properties, line 35
> > 
> >
> > Hmm, are we doing this?

had to put it in as there was just too much noise coming from the hive 
libraries during normal flume operation.


- Roshan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/#review51208
---


On Sept. 3, 2014, 3:19 a.m., Roshan Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18544/
> ---
> 
> (Updated Sept. 3, 2014, 3:19 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-1734
> https://issues.apache.org/jira/browse/FLUME-1734
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Hive streaming sink.
> 
> 
> Diffs
> -
> 
>   bin/flume-ng e09e26b 
>   conf/log4j.properties 3918511 
>   
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java
>  ac11558 
>   
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 
> 0a1cd7a 
>   flume-ng-dist/pom.xml 8c18af6 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst a718fbf 
>   
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java
>  ff32c45 
>   
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java
>  8e08f22 
>   
> flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 
> 7f966b0 
>   flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/Config.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java
>  PRE-CREATION 
>   flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties 
> PRE-CREATION 
>   flume-ng-sinks/pom.xml 3381bde 
>   
> flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java
>  eba8d2e 
>   pom.xml 4bdfcac 
> 
> Diff: https://reviews.apache.org/r/18544/diff/
> 
> 
> Testing
> ---
> 
> includes unit tests.
> 
> 
> Thanks,
> 
> Roshan Naik
> 
>

[jira] [Updated] (FLUME-2549) Enable SSLv2Hello for HttpSource

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2549:

Attachment: FLUME-2549.patch

> Enable SSLv2Hello for HttpSource
> 
>
> Key: FLUME-2549
> URL: https://issues.apache.org/jira/browse/FLUME-2549
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Hari Shreedharan
> Attachments: FLUME-2549.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2548) Enable SSLv2Hello for Avro Source and NettyAvroRpcClient

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2548:

Attachment: FLUME-2549.patch

> Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
> 
>
> Key: FLUME-2548
> URL: https://issues.apache.org/jira/browse/FLUME-2548
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2548.patch, FLUME-2549.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2548) Enable SSLv2Hello for Avro Source and NettyAvroRpcClient

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2548:

Attachment: (was: FLUME-2549.patch)

> Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
> 
>
> Key: FLUME-2548
> URL: https://issues.apache.org/jira/browse/FLUME-2548
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2548.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2548) Enable SSLv2Hello for Avro Source and NettyAvroRpcClient

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2548:

Attachment: FLUME-2548.patch

> Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
> 
>
> Key: FLUME-2548
> URL: https://issues.apache.org/jira/browse/FLUME-2548
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2548.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2548) Enable SSLv2 for Avro Source and NettyAvroRpcClient

[Hari Shreedharan (JIRA)]

Hari Shreedharan created FLUME-2548:
---

 Summary: Enable SSLv2 for Avro Source and NettyAvroRpcClient
 Key: FLUME-2548
 URL: https://issues.apache.org/jira/browse/FLUME-2548
 Project: Flume
  Issue Type: Sub-task
Reporter: Hari Shreedharan
Assignee: Hari Shreedharan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2549) Enable SSLv2Hello for HttpSource

[Hari Shreedharan (JIRA)]

Hari Shreedharan created FLUME-2549:
---

 Summary: Enable SSLv2Hello for HttpSource
 Key: FLUME-2549
 URL: https://issues.apache.org/jira/browse/FLUME-2549
 Project: Flume
  Issue Type: Sub-task
Reporter: Hari Shreedharan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2547) Removing SSLv2Hello causes Java 6 clients to break

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2547:

Attachment: (was: FLUME-2547.patch)

> Removing SSLv2Hello causes Java 6 clients to break
> --
>
> Key: FLUME-2547
> URL: https://issues.apache.org/jira/browse/FLUME-2547
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
>
> In Java 6, if the server side does not accept SSLv3, SSLv2Hello is required 
> even if TLS is used. SSLv2Hello itself is not insecure so we should bring it 
> back for compat with older Java versions



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2548) Enable SSLv2Hello for Avro Source and NettyAvroRpcClient

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2548:

Summary: Enable SSLv2Hello for Avro Source and NettyAvroRpcClient  (was: 
Enable SSLv2 for Avro Source and NettyAvroRpcClient)

> Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
> 
>
> Key: FLUME-2548
> URL: https://issues.apache.org/jira/browse/FLUME-2548
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2547) Removing SSLv2Hello causes Java 6 clients to break

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2547:

Attachment: FLUME-2547.patch

> Removing SSLv2Hello causes Java 6 clients to break
> --
>
> Key: FLUME-2547
> URL: https://issues.apache.org/jira/browse/FLUME-2547
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2547.patch
>
>
> In Java 6, if the server side does not accept SSLv3, SSLv2Hello is required 
> even if TLS is used. SSLv2Hello itself is not insecure so we should bring it 
> back for compat with older Java versions



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2546) HTTP Source docs should not that neither SSLv2Hello nor SSLv3 are supported

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207231#comment-14207231
 ] 

Hari Shreedharan commented on FLUME-2546:
-

Holding off on this one till FLUME-2547 is done.

> HTTP Source docs should not that neither SSLv2Hello nor SSLv3 are supported
> ---
>
> Key: FLUME-2546
> URL: https://issues.apache.org/jira/browse/FLUME-2546
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2546.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

SSLv2Hello is required on Java 6

[Hari Shreedharan]

Hi,


Right after we pushed 1.5.1 out (I have not even sent the announce email), we 
discovered that Java 6 requires SSLv2Hello on the server side for negotiation 
even if TLS is used (unless the client code is also changed to disable 
SSLv2Hello). So:


- For HTTP Source, any clients running on Java 6 would need code changes to 
also disable SSLv2Hello to be able to send data via TLSv1.
- For Avro Source, any clients running Flume SDK < 1.5.1 on Java 6 would break 
and requires the client application to upgrade to 1.5.1. 


I filed FLUME-2547 to fix this.


My question to the community here is whether we want a new release bringing 
SSLv2Hello back or if we are willing to just document this and move forward?


I am willing to put together an RC if required.

Thanks,
Hari

[jira] [Created] (FLUME-2547) Removing SSLv2Hello causes Java 6 clients to break

[Hari Shreedharan (JIRA)]

Hari Shreedharan created FLUME-2547:
---

 Summary: Removing SSLv2Hello causes Java 6 clients to break
 Key: FLUME-2547
 URL: https://issues.apache.org/jira/browse/FLUME-2547
 Project: Flume
  Issue Type: Bug
Reporter: Hari Shreedharan
Assignee: Hari Shreedharan


In Java 6, if the server side does not accept SSLv3, SSLv2Hello is required 
even if TLS is used. SSLv2Hello itself is not insecure so we should bring it 
back for compat with older Java versions



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2546) HTTP Source docs should not that neither SSLv2Hello nor SSLv3 are supported

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2546:

Attachment: FLUME-2546.patch

I am applying this patch to the 1.5.1 docs and uploading to the website

> HTTP Source docs should not that neither SSLv2Hello nor SSLv3 are supported
> ---
>
> Key: FLUME-2546
> URL: https://issues.apache.org/jira/browse/FLUME-2546
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.6.0
>
> Attachments: FLUME-2546.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2546) HTTP Source docs should not that neither SSLv2Hello nor SSLv3 are supported

[Hari Shreedharan (JIRA)]

Hari Shreedharan created FLUME-2546:
---

 Summary: HTTP Source docs should not that neither SSLv2Hello nor 
SSLv3 are supported
 Key: FLUME-2546
 URL: https://issues.apache.org/jira/browse/FLUME-2546
 Project: Flume
  Issue Type: Bug
  Components: Sinks+Sources
Affects Versions: v1.5.1
Reporter: Hari Shreedharan
Assignee: Hari Shreedharan
 Fix For: v1.6.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Flume 1.5.1 RC1

[Roshan Naik]

Thanks Hari for all the work.
-roshan

On Mon, Nov 10, 2014 at 10:55 PM, Hari Shreedharan <
hshreedha...@cloudera.com> wrote:

> This vote is now closed.
>
>
>
>
> This vote has passed with 4 +1 binding votes:
>
> - Arvind Prabhakar
>
> - Roshan Naik
>
> - Hari Shreedharan
>
> - Jarcec
>
>
>
>
> Non-binding +1 votes:
>
> - Ashish Paliwal
>
>
>
>
> No +0 or -1 votes were received.
>
>
>
>
> As a result, this RC has passed and will be promoted to the Apache Flume
> 1.5.1 release. I will send out an email announcing the release soon!
>
>
>
>
>
> Thanks,
> Hari
>
> On Mon, Nov 10, 2014 at 9:33 PM, Jarek Jarcec Cecho 
> wrote:
>
> > +1
> > * Verified checksums and signature files
> > * Verified that each jar in binary tarball is in the license
> > * Checked top level files (NOTICE, ...)
> > * Run tests
> > Jarcec
> >> On Nov 6, 2014, at 3:17 PM, Hari Shreedharan 
> wrote:
> >>
> >> This is the seventh release for Apache Flume as a top-level project,
> >> version 1.5.1. We are voting on release candidate RC1.
> >>
> >> It fixes the following issues:
> >>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920
> >>
> >> *** Please cast your vote within the next 72 hours ***
> >>
> >> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
> >> for the source and binary artifacts can be found here:
> >>  https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/
> >>
> >> Maven staging repo:
> >>
> https://repository.apache.org/content/repositories/orgapacheflume-1006/
> >>
> >> The tag to be voted on:
> >>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920
> >>
> >> Flume's KEYS file containing PGP keys we use to sign the release:
> >>  http://www.apache.org/dist/flume/KEYS
> >>
> >> Thanks,
> >> Hari
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (FLUME-2545) How to convert *.bz2.tmp to *.bz2 file after restating the instance

[Arun Gujjar (JIRA)]

Arun Gujjar created FLUME-2545:
--

 Summary: How to convert *.bz2.tmp to *.bz2 file after restating 
the instance
 Key: FLUME-2545
 URL: https://issues.apache.org/jira/browse/FLUME-2545
 Project: Flume
  Issue Type: Bug
  Components: Sinks+Sources
Affects Versions: v1.4.0
Reporter: Arun Gujjar
Priority: Critical


Hi,

Whenever we restart flume agent it creates a new HDFS file and start writing 
the data into that file. The earlier file which was created will still be left 
as *bz2.tmp and from HIVE queries we found that we were not able to read the 
data from this file. 

Here are the two questions I have .
1. Could you please suggest how we can convert this bz2.tmp to bz2 file? 
because we loose this data i.e. present in bz2.tmp file today. 
2. Is there as way to configure flume to start writing the data into the 
existing bz2.tmp file instead of creating a new file?
3. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2441) Unit test TestHTTPSource.java failed with IBM JDK 1.7

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206932#comment-14206932
 ] 

Hari Shreedharan commented on FLUME-2441:
-

It went into 1.5.1

> Unit test TestHTTPSource.java failed with IBM JDK 1.7
> -
>
> Key: FLUME-2441
> URL: https://issues.apache.org/jira/browse/FLUME-2441
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources, Test
>Affects Versions: v1.5.0.1, v1.6.0
> Environment: RHEL server 6.3
> IBM JDK 1.7.0
>Reporter: li xiang
>Assignee: li xiang
>Priority: Minor
>  Labels: test
> Fix For: v1.5.1
>
> Attachments: FLUME-2441.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> TestHTTPSource.java (under 
> flume-ng-core/src/test/java/org/apache/flume/source/http) failed with IBM JDK 
> 1.7. Oracle JDK 1.7 does not have this failure.
> ---
> Test set: org.apache.flume.source.http.TestHTTPSource
> ---
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.535 sec 
> <<< FAILURE!
> testHttps(org.apache.flume.source.http.TestHTTPSource)  Time elapsed: 168 sec 
>  <<< FAILURE!
> junit.framework.AssertionFailedError: Exception not expected
>   at junit.framework.Assert.fail(Assert.java:50)
>   at 
> org.apache.flume.source.http.TestHTTPSource.testHttps(TestHTTPSource.java:388)
> The root exception is:
> javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
>   at com.ibm.jsse2.j.a(j.java:4)
>   at com.ibm.jsse2.j.a(j.java:31)
>   at com.ibm.jsse2.qc.b(qc.java:624)
>   at com.ibm.jsse2.qc.a(qc.java:850)
>   at com.ibm.jsse2.qc.h(qc.java:194)
>   at com.ibm.jsse2.qc.a(qc.java:477)
>   at com.ibm.jsse2.qc.startHandshake(qc.java:672)
>   at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:87)
>   at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:21)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1103)
>   at com.ibm.net.ssl.www2.protocol.https.b.getOutputStream(b.java:75)
>   at 
> org.apache.flume.source.http.TestHTTPSource.testHttps(TestHTTPSource.java:372)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)2014-08-04 01:56:40,263 (-1175282339@qtp--597224782-0) [WARN - 
> org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:89)] EXCEPTION 
> javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 not 
> enabled or not supported
>   at com.ibm.jsse2.j.a(j.java:4)
>   at com.ibm.jsse2.qc.a(qc.java:513)
>   at com.ibm.jsse2.ab.a(ab.java:385)
>   at com.ibm.jsse2.ab.a(ab.java:96)
>   at com.ibm.jsse2.cb.a(cb.java:501)
>   at com.ibm.jsse2.cb.a(cb.java:409)
>   at com.ibm.jsse2.ab.r(ab.java:489)
>   at com.ibm.jsse2.ab.a(ab.java:65)
>   at com.ibm.jsse2.qc.a(qc.java:359)
>   at com.ibm.jsse2.qc.h(qc.java:194)
>   at com.ibm.jsse2.qc.a(qc.java:477)
>   at com.ibm.jsse2.qc.startHandshake(qc.java:672)
>   at 
> org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
>   at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
>   at java.lang.reflect.Method.invoke(Method.java:619)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at 
> org.junit.internal.runners.statement

[jira] [Commented] (FLUME-2540) FLUME failed to rotate HDFS sink file from *.tmp to normal when timezone changed from PDT - PST on Nov 02, 2014

[Arun Gujjar (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206928#comment-14206928
 ] 

Arun Gujjar commented on FLUME-2540:


Thanks Roshan, I thought of trying this option, will try and let you know. 

> FLUME failed to rotate HDFS sink file from *.tmp to normal when timezone 
> changed from PDT - PST on Nov 02, 2014
> ---
>
> Key: FLUME-2540
> URL: https://issues.apache.org/jira/browse/FLUME-2540
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.4.0
> Environment: apache-flume-1.4.0-cdh4.5.0, zookeeper-3.4.5-cdh4.5.0
>Reporter: Arun Gujjar
>
> I have 2 instance of FLUME installed in a given host which has a timezone set 
> in GMT. The day when time changed from PDT - PST on Nov 02, 2014. One of the 
> flume instance failed to convert sink FlumeData.x.bz2.tmp to a 
> FlumeData.x.bz2 file, but the other instance had no issues of this 
> timezone change. 
> The difference between these two instances is, the instance which failed to 
> convert sink filename writes the data ~6mb per hour, but other instance 
> writes ~100 kb per hour.
> Could you please suggest what could be the reason and how to convert file 
> from FlumeData.x.bz2.tmp to FlumeData.x.bz2 file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Using CodaHale metrics for monitoring

[Hari Shreedharan]

I think it is best to replace the current system rather than having multiple 
metrics systems. We should implement it such that it is transparent to user 
code, and the changes only go into MonitoredCounterGroup (the CounterGroup 
class is obsolete - dont bother).


Thanks,
Hari

On Mon, Nov 10, 2014 at 5:36 PM, Ashish  wrote:

> Codahale does have lot more features like the rate/sec one, which is
> most needed metrics. It provides a lot of reporters out of box
> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
> enable it, rather than writing custom components.
> As of now we have to hide it within MonitoredCounterGroup and its
> sub-classes, that's what I did for running it. It was running parallel
> to existing system. Let me work more on it and see if I can simplify
> the same.
> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>  wrote:
>> Is it easier to use than the current one and/or does it give better 
>> performance? You’d need to support the current metrics API 
>> (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>
>>
>> Thanks,
>> Hari
>>
>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish  wrote:
>>
>>> Hi,
>>> Have hacked a bit into our existing instrumentation package and piggy
>>> backed cohahale metrics package. Here is one sample for Spooled
>>> Directory source (with instrumentation only for Source and Channel ),
>>> using console reporter
>>> -- Gauges 
>>> --
>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>>  value = 200
>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>>  value = 2.0
>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>>  value = 0
>>> -- Counters 
>>> 
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>>  count = 1138800
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>>  count = 1138800
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>>  count = 1138601
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>>  count = 1138600
>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>>  count = 1138800
>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>>  count = 1138800
>>> src.append-batch.accepted
>>>  count = 11388
>>> src.append-batch.received
>>>  count = 11388
>>> src.append.accepted
>>>  count = 0
>>> src.append.received
>>>  count = 0
>>> -- Meters 
>>> --
>>> eventAcceptedMeter
>>>  count = 1138800
>>>  mean rate = 106478.56 events/second
>>>  1-minute rate = 93420.18 events/second
>>>  5-minute rate = 91389.24 events/second
>>> 15-minute rate = 91037.40 events/second
>>> eventReceivedMeter
>>>  count = 1138800
>>>  mean rate = 106462.14 events/second
>>>  1-minute rate = 93420.18 events/second
>>>  5-minute rate = 91389.24 events/second
>>> 15-minute rate = 91037.40 events/second
>>> If there is interest in the community, can raise a jira and continue
>>> to work on it.
>>> --
>>> thanks
>>> ashish
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
> -- 
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal

[jira] [Updated] (FLUME-2516) TestElasticSearchIndexRequestBuilderFactory fails with Locale != ENGLISH

[Santiago M. Mola (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santiago M. Mola updated FLUME-2516:

Labels: easyfix easytest  (was: )

> TestElasticSearchIndexRequestBuilderFactory fails with Locale != ENGLISH
> 
>
> Key: FLUME-2516
> URL: https://issues.apache.org/jira/browse/FLUME-2516
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Reporter: Santiago M. Mola
>Assignee: Santiago M. Mola
>  Labels: easyfix, easytest
> Fix For: v1.6.0
>
> Attachments: FLUME-2516-0.patch
>
>
> TestElasticSearchIndexRequestBuilderFactory fails when Locale != ENGLISH.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2487) SyslogParser has rounding errors in timestamp parsing

[Santiago M. Mola (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santiago M. Mola updated FLUME-2487:

Summary: SyslogParser has rounding errors in timestamp parsing  (was: 
Timestamp mismatch in TestMultiportSyslogTCPSource)

> SyslogParser has rounding errors in timestamp parsing
> -
>
> Key: FLUME-2487
> URL: https://issues.apache.org/jira/browse/FLUME-2487
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: Santiago M. Mola
>Assignee: Santiago M. Mola
>  Labels: test-fail
> Fix For: v1.6.0
>
> Attachments: FLUME-2487-0.patch
>
>
> https://travis-ci.org/Stratio/flume/jobs/36847695#L6180
> {noformat}
> testMultiplePorts(org.apache.flume.source.TestMultiportSyslogTCPSource) Time 
> elapsed: 2465 sec <<< FAILURE!
> org.junit.ComparisonFailure: Timestamps must match expected:<141223934050[2]> 
> but was:<141223934050[1]>
> at org.junit.Assert.assertEquals(Assert.java:125)
> at 
> org.apache.flume.source.TestMultiportSyslogTCPSource.testMultiplePorts(TestMultiportSyslogTCPSource.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2487) Timestamp mismatch in TestMultiportSyslogTCPSource

[Santiago M. Mola (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santiago M. Mola updated FLUME-2487:

Attachment: FLUME-2487-0.patch

Patch updated with test case that may prove the bug.

> Timestamp mismatch in TestMultiportSyslogTCPSource
> --
>
> Key: FLUME-2487
> URL: https://issues.apache.org/jira/browse/FLUME-2487
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: Santiago M. Mola
>Assignee: Santiago M. Mola
>  Labels: test-fail
> Fix For: v1.6.0
>
> Attachments: FLUME-2487-0.patch
>
>
> https://travis-ci.org/Stratio/flume/jobs/36847695#L6180
> {noformat}
> testMultiplePorts(org.apache.flume.source.TestMultiportSyslogTCPSource) Time 
> elapsed: 2465 sec <<< FAILURE!
> org.junit.ComparisonFailure: Timestamps must match expected:<141223934050[2]> 
> but was:<141223934050[1]>
> at org.junit.Assert.assertEquals(Assert.java:125)
> at 
> org.apache.flume.source.TestMultiportSyslogTCPSource.testMultiplePorts(TestMultiportSyslogTCPSource.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2487) Timestamp mismatch in TestMultiportSyslogTCPSource

[Santiago M. Mola (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santiago M. Mola updated FLUME-2487:

Attachment: (was: FLUME-2487-0.patch)

> Timestamp mismatch in TestMultiportSyslogTCPSource
> --
>
> Key: FLUME-2487
> URL: https://issues.apache.org/jira/browse/FLUME-2487
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: Santiago M. Mola
>Assignee: Santiago M. Mola
>  Labels: test-fail
> Fix For: v1.6.0
>
>
> https://travis-ci.org/Stratio/flume/jobs/36847695#L6180
> {noformat}
> testMultiplePorts(org.apache.flume.source.TestMultiportSyslogTCPSource) Time 
> elapsed: 2465 sec <<< FAILURE!
> org.junit.ComparisonFailure: Timestamps must match expected:<141223934050[2]> 
> but was:<141223934050[1]>
> at org.junit.Assert.assertEquals(Assert.java:125)
> at 
> org.apache.flume.source.TestMultiportSyslogTCPSource.testMultiplePorts(TestMultiportSyslogTCPSource.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2487) Timestamp mismatch in TestMultiportSyslogTCPSource

[Santiago M. Mola (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206507#comment-14206507
 ] 

Santiago M. Mola commented on FLUME-2487:
-

[~ashishpaliwal] Given the non-deterministic nature of the test (dependent of 
the time its run), chances are that you never get this error running the test. 
But the bug is clearly visible in the code: floating point numbers are not 
exact and they lead to rounding errors. This is a well-known fact. Just run 
this yourself:

  @Test
  public void testXXX() {
for (long i = 100; i < 999; i++) {
  float frac = Float.parseFloat(String.format(".%d", i));
  float multiplied = (frac * 1000f);
  long milliseconds = (long) multiplied;
  if (i != milliseconds) {
System.out.println("PRECISION ERROR: " + i + " | " + milliseconds + " | 
" + frac + " | " + multiplied);
  }
}
  }

Output:
PRECISION ERROR: 251 | 250 | 0.251 | 250.8
PRECISION ERROR: 253 | 252 | 0.253 | 252.8
PRECISION ERROR: 502 | 501 | 0.502 | 501.7
PRECISION ERROR: 506 | 505 | 0.506 | 505.7
PRECISION ERROR: 511 | 510 | 0.511 | 510.7


This code needs to use integer arithmetic or BigDecimal. My patch uses just 
integer arithmetic, but I can write one to use BigDecimal if that's preferred.

> Timestamp mismatch in TestMultiportSyslogTCPSource
> --
>
> Key: FLUME-2487
> URL: https://issues.apache.org/jira/browse/FLUME-2487
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: Santiago M. Mola
>Assignee: Santiago M. Mola
>  Labels: test-fail
> Fix For: v1.6.0
>
> Attachments: FLUME-2487-0.patch
>
>
> https://travis-ci.org/Stratio/flume/jobs/36847695#L6180
> {noformat}
> testMultiplePorts(org.apache.flume.source.TestMultiportSyslogTCPSource) Time 
> elapsed: 2465 sec <<< FAILURE!
> org.junit.ComparisonFailure: Timestamps must match expected:<141223934050[2]> 
> but was:<141223934050[1]>
> at org.junit.Assert.assertEquals(Assert.java:125)
> at 
> org.apache.flume.source.TestMultiportSyslogTCPSource.testMultiplePorts(TestMultiportSyslogTCPSource.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2441) Unit test TestHTTPSource.java failed with IBM JDK 1.7

[li xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206393#comment-14206393
 ] 

li xiang commented on FLUME-2441:
-

Hi Hari, this fix also goes into flume-1.5 ?

> Unit test TestHTTPSource.java failed with IBM JDK 1.7
> -
>
> Key: FLUME-2441
> URL: https://issues.apache.org/jira/browse/FLUME-2441
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources, Test
>Affects Versions: v1.5.0.1, v1.6.0
> Environment: RHEL server 6.3
> IBM JDK 1.7.0
>Reporter: li xiang
>Assignee: li xiang
>Priority: Minor
>  Labels: test
> Fix For: v1.5.1
>
> Attachments: FLUME-2441.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> TestHTTPSource.java (under 
> flume-ng-core/src/test/java/org/apache/flume/source/http) failed with IBM JDK 
> 1.7. Oracle JDK 1.7 does not have this failure.
> ---
> Test set: org.apache.flume.source.http.TestHTTPSource
> ---
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.535 sec 
> <<< FAILURE!
> testHttps(org.apache.flume.source.http.TestHTTPSource)  Time elapsed: 168 sec 
>  <<< FAILURE!
> junit.framework.AssertionFailedError: Exception not expected
>   at junit.framework.Assert.fail(Assert.java:50)
>   at 
> org.apache.flume.source.http.TestHTTPSource.testHttps(TestHTTPSource.java:388)
> The root exception is:
> javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
>   at com.ibm.jsse2.j.a(j.java:4)
>   at com.ibm.jsse2.j.a(j.java:31)
>   at com.ibm.jsse2.qc.b(qc.java:624)
>   at com.ibm.jsse2.qc.a(qc.java:850)
>   at com.ibm.jsse2.qc.h(qc.java:194)
>   at com.ibm.jsse2.qc.a(qc.java:477)
>   at com.ibm.jsse2.qc.startHandshake(qc.java:672)
>   at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:87)
>   at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:21)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1103)
>   at com.ibm.net.ssl.www2.protocol.https.b.getOutputStream(b.java:75)
>   at 
> org.apache.flume.source.http.TestHTTPSource.testHttps(TestHTTPSource.java:372)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)2014-08-04 01:56:40,263 (-1175282339@qtp--597224782-0) [WARN - 
> org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:89)] EXCEPTION 
> javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 not 
> enabled or not supported
>   at com.ibm.jsse2.j.a(j.java:4)
>   at com.ibm.jsse2.qc.a(qc.java:513)
>   at com.ibm.jsse2.ab.a(ab.java:385)
>   at com.ibm.jsse2.ab.a(ab.java:96)
>   at com.ibm.jsse2.cb.a(cb.java:501)
>   at com.ibm.jsse2.cb.a(cb.java:409)
>   at com.ibm.jsse2.ab.r(ab.java:489)
>   at com.ibm.jsse2.ab.a(ab.java:65)
>   at com.ibm.jsse2.qc.a(qc.java:359)
>   at com.ibm.jsse2.qc.h(qc.java:194)
>   at com.ibm.jsse2.qc.a(qc.java:477)
>   at com.ibm.jsse2.qc.startHandshake(qc.java:672)
>   at 
> org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
>   at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
>   at java.lang.reflect.Method.invoke(Method.java:619)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at 
> org.junit.internal.runners

Jenkins build is back to stable : Flume-trunk-hbase-98 #50

[Apache Jenkins Server]

See 

Jenkins build is still unstable: flume-trunk #693

[Apache Jenkins Server]

See 

[jira] [Commented] (FLUME-2385) Flume spans log file with "Spooling Directory Source runner has shutdown" messages at INFO level

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206103#comment-14206103
 ] 

Hudson commented on FLUME-2385:
---

UNSTABLE: Integrated in flume-trunk #692 (See 
[https://builds.apache.org/job/flume-trunk/692/])
FLUME-2385. Remove incorrect log message at INFO level in Spool Directory 
Source. (hshreedharan: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=2c18533253be786b9c60bf687cdf38d2384d2625)
* flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java


> Flume spans log file with "Spooling Directory Source runner has shutdown" 
> messages at INFO level
> 
>
> Key: FLUME-2385
> URL: https://issues.apache.org/jira/browse/FLUME-2385
> Project: Flume
>  Issue Type: Improvement
>Affects Versions: v1.4.0
>Reporter: Justin Hayes
>Assignee: Phil Scala
>Priority: Minor
> Fix For: v1.6.0
>
> Attachments: FLUME-2385-0.patch
>
>
> When I start an agent with the following config, the spooling directory 
> source emits "14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling 
> Directory Source runner has shutdown." messages twice a second. Pretty 
> innocuous but it will fill up the file system needlessly and get in the way 
> of other INFO messages.
> cis.sources = httpd
> cis.sinks = loggerSink
> cis.channels = mem2logger
> cis.sources.httpd.type = spooldir
> cis.sources.httpd.spoolDir = /var/log/httpd
> cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd
> cis.sources.httpd.channels = mem2logger
> cis.sinks.loggerSink.type = logger
> cis.sinks.loggerSink.channel = mem2logger
> cis.channels.mem2logger.type = memory
> cis.channels.mem2logger.capacity = 1
> cis.channels.mem2logger.transactionCapacity = 1000 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: flume-trunk #692

[Apache Jenkins Server]

See