Re: Flume+ML [Discussion]

[Johny Rufus John]

Hi Saikat,

Have you considered this approach,

http://spark.apache.org/docs/latest/streaming-flume-integration.html



Thanks,
Johny

On Thu, Feb 2, 2017 at 5:27 PM, Saikat Kanjilal  wrote:

> Hi Flume community,
> Would love to have inputs on this topic as this is a pertinent usecase
> that I'm exploring at work.
> Regards
>
> Sent from my iPhone
>
> On Feb 1, 2017, at 4:36 PM, Saikat Kanjilal mailto:sx
> k1...@hotmail.com>> wrote:
>
>
> Bump [😊]
>
>
> 
> From: Saikat Kanjilal mailto:sxk1...@hotmail.com>>
> Sent: Wednesday, February 1, 2017 8:46 AM
> To: dev@flume.apache.org
> Subject: Flume+ML [Discussion]
>
> Hi Folks,
>
> I've been a bit delayed working on the graph sink for flume (
> https://github.com/skanjila/flume-ng-graphstore-sink) , in the meantime I
> was wondering if there's been any thought or interest in connecting flume
> to spark, I have a potential use case where we need to extract data out of
> multiple data sources, do a set of transformations on this data and then
> dump this data to a columnar store for downstream processing through a
> Revoscale R cluster which uses spark underneath.  I'd be interested in
> leading this effort if there's enough interest in the community around use
> cases for this.
> [https://avatars0.githubusercontent.com/u/674374?v=3&s=400] github.com/skanjila/flume-ng-graphstore-sink>
>
> skanjila/flume-ng-graphstore-sink skanjila/flume-ng-graphstore-sink>
> github.com
> flume-ng-graphstore-sink - A flume sink that writes to a set of graph
> databases
>
>
>
>
> [https://avatars0.githubusercontent.com/u/674374?v=3&s=400] github.com/skanjila/flume-ng-graphstore-sink>
>
> skanjila/flume-ng-graphstore-sink skanjila/flume-ng-graphstore-sink>
> github.com
> flume-ng-graphstore-sink - A flume sink that writes to a set of graph
> databases
>
> Look forward to hearing from folks.
>

Re: [ANNOUNCE] Two new Flume committers

[Johny Rufus John]

Congrats Bessenyei and Jeff !!

Regards,
Rufus

On Mon, Sep 19, 2016 at 4:43 PM, Mike Percy  wrote:

> Hi Apache Flume community,
>
> I am very happy to announce that the Flume PMC has voted to add Bessenyei
> Balázs Donát and Jeff Holoman as committers in recognition of their
> contributions to Flume.
>
> Over the past few months, Donat has contributed and reviewed many patches,
> more than any non-committer. He has contributed several bug fixes and
> improvements and has shepherded important, long-forgotten patches through
> the review and commit process, with more  in-progress. He is also currently
> working on improvements to the Flume configuration system.
>
> Jeff has contributed several important improvements to Flume in recent
> months, including adding support for secure Kafka to Flume, improving the
> AvroEventSerializer, and adding additional smarts to the HDFS sink.
>
> Please join me in congratulating them on their new committership!
>
> Best regards,
> Mike
>
>

[jira] [Commented] (FLUME-2437) S3 Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436907#comment-15436907
 ] 

Johny Rufus commented on FLUME-2437:


[~David KIM], Am in the process of getting a new patch that we can submit, can 
you paste the error you were getting and the relevant Flume log entries if 
possible, to see more on the issue you were facing

> S3 Source
> -
>
> Key: FLUME-2437
> URL: https://issues.apache.org/jira/browse/FLUME-2437
> Project: Flume
>  Issue Type: New Feature
>Reporter: Jonathan Natkins
>    Assignee: Johny Rufus
> Fix For: v2.0.0
>
> Attachments: FLUME-2437-2.patch, FLUME-2437.patch
>
>
> There have been multiple requests on the mailing list for an S3 source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2437) S3 Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424647#comment-15424647
 ] 

Johny Rufus commented on FLUME-2437:


[~bessbd], Sure, let me work on this, by this weekend, rebase and run the tests 
and upload the new patch.

> S3 Source
> -
>
> Key: FLUME-2437
> URL: https://issues.apache.org/jira/browse/FLUME-2437
> Project: Flume
>  Issue Type: New Feature
>Reporter: Jonathan Natkins
>    Assignee: Johny Rufus
> Fix For: v2.0.0
>
> Attachments: FLUME-2437-2.patch, FLUME-2437.patch
>
>
> There have been multiple requests on the mailing list for an S3 source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2912) thrift Sources/Sinks can only authenticate with kerberos principal in format with hostname

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307931#comment-15307931
 ] 

Johny Rufus commented on FLUME-2912:


[~wpwang], I will take a look at this.

> thrift Sources/Sinks can only authenticate with kerberos principal in  format 
> with hostname
> ---
>
> Key: FLUME-2912
> URL: https://issues.apache.org/jira/browse/FLUME-2912
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Ping Wang
>Assignee: Johny Rufus
> Fix For: v1.7.0
>
>
> Using Thrift Sources/Sinks in Kerberos environment, the Flume agents
> only work with principle in format "name/_h...@your-realm.com".  
> If using other valid principle in the format "n...@your-realm.com"  it will 
> hit ERROR of "GSS initiate failed".  
> Here's the configuration file:
> g1.sources.source1.type = spooldir
> g1.sources.source1.spoolDir = /test
> g1.sources.source1.fileHeader = false
> g1.sinks.sink1.type = thrift
> g1.sinks.sink1.hostname = localhost
> g1.sinks.sink1.port = 5
> g1.channels.channel1.type = memory
> g1.channels.channel1.capacity = 1000
> g1.channels.channel1.transactionCapacity = 100
> g1.sources.source1.channels = channel1
> g1.sinks.sink1.channel = channel1
> g2.sources = source2
> g2.sinks = sink2
> g2.channels = channel2
> g2.sources.source2.type = thrift
> g2.sources.source2.bind = localhost
> g2.sources.source2.port = 5
> g2.sinks.sink2.type = hdfs
> g2.sinks.sink2.hdfs.path = /tmp
> g2.sinks.sink2.hdfs.filePrefix = thriftData
> g2.sinks.sink2.hdfs.writeFormat = Text
> g2.sinks.sink2.hdfs.fileType = DataStream
> g2.channels.channel2.type = memory
> g2.channels.channel2.capacity = 1000
> g2.channels.channel2.transactionCapacity = 100
> g2.sources.source2.channels = channel2
> g2.sinks.sink2.channel = channel2
> g1.sinks.sink1.kerberos = true
> g1.sinks.sink1.client-principal = flume/hostn...@xxx.com
> g1.sinks.sink1.client-keytab
> = /etc/security/keytabs/flume-1563.server.keytab
> g1.sinks.sink1.server-principal = flume/hostn...@xxx.com
> g2.sources.source2.kerberos = true
> g2.sources.source2.agent-principal = flume/hostn...@xxx.com
> g2.sources.source2.agent-keytab
> = /etc/security/keytabs/flume-1563.server.keytab
> If using other valid principle like "t...@ibm.com" as below, will hit error:
> g1.sinks.sink1.kerberos = true
> g1.sinks.sink1.client-principal = t...@ibm.com
> g1.sinks.sink1.client-keytab = /home/test/test.keytab
> g1.sinks.sink1.server-principal = t...@ibm.com
> g2.sources.source2.kerberos = true
> g2.sources.source2.agent-principal = t...@ibm.com
> g2.sources.source2.agent-keytab = /home/test/test.keytab
> Agent g1:
> ERROR server.TThreadPoolServer: Error occurred during processing of
> message.
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException: Peer indicated failure:
> GSS initiate failed
>     at org.apache.thrift.transport.TSaslServerTransport
> $Factory.getTransport(TSaslServerTransport.java:219)
>     at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run
> (TThreadPoolServer.java:189)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1142)
> Agent g2:
> ERROR transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - UNKNOWN_SERVER)]
>     at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge
> (GssKrb5Client.java:211)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLUME-2912) thrift Sources/Sinks can only authenticate with kerberos principal in format with hostname

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus reassigned FLUME-2912:
--

Assignee: Johny Rufus

> thrift Sources/Sinks can only authenticate with kerberos principal in  format 
> with hostname
> ---
>
> Key: FLUME-2912
> URL: https://issues.apache.org/jira/browse/FLUME-2912
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Ping Wang
>Assignee: Johny Rufus
> Fix For: v1.7.0
>
>
> Using Thrift Sources/Sinks in Kerberos environment, the Flume agents
> only work with principle in format "name/_h...@your-realm.com".  
> If using other valid principle in the format "n...@your-realm.com"  it will 
> hit ERROR of "GSS initiate failed".  
> Here's the configuration file:
> g1.sources.source1.type = spooldir
> g1.sources.source1.spoolDir = /test
> g1.sources.source1.fileHeader = false
> g1.sinks.sink1.type = thrift
> g1.sinks.sink1.hostname = localhost
> g1.sinks.sink1.port = 5
> g1.channels.channel1.type = memory
> g1.channels.channel1.capacity = 1000
> g1.channels.channel1.transactionCapacity = 100
> g1.sources.source1.channels = channel1
> g1.sinks.sink1.channel = channel1
> g2.sources = source2
> g2.sinks = sink2
> g2.channels = channel2
> g2.sources.source2.type = thrift
> g2.sources.source2.bind = localhost
> g2.sources.source2.port = 5
> g2.sinks.sink2.type = hdfs
> g2.sinks.sink2.hdfs.path = /tmp
> g2.sinks.sink2.hdfs.filePrefix = thriftData
> g2.sinks.sink2.hdfs.writeFormat = Text
> g2.sinks.sink2.hdfs.fileType = DataStream
> g2.channels.channel2.type = memory
> g2.channels.channel2.capacity = 1000
> g2.channels.channel2.transactionCapacity = 100
> g2.sources.source2.channels = channel2
> g2.sinks.sink2.channel = channel2
> g1.sinks.sink1.kerberos = true
> g1.sinks.sink1.client-principal = flume/hostn...@xxx.com
> g1.sinks.sink1.client-keytab
> = /etc/security/keytabs/flume-1563.server.keytab
> g1.sinks.sink1.server-principal = flume/hostn...@xxx.com
> g2.sources.source2.kerberos = true
> g2.sources.source2.agent-principal = flume/hostn...@xxx.com
> g2.sources.source2.agent-keytab
> = /etc/security/keytabs/flume-1563.server.keytab
> If using other valid principle like "t...@ibm.com" as below, will hit error:
> g1.sinks.sink1.kerberos = true
> g1.sinks.sink1.client-principal = t...@ibm.com
> g1.sinks.sink1.client-keytab = /home/test/test.keytab
> g1.sinks.sink1.server-principal = t...@ibm.com
> g2.sources.source2.kerberos = true
> g2.sources.source2.agent-principal = t...@ibm.com
> g2.sources.source2.agent-keytab = /home/test/test.keytab
> Agent g1:
> ERROR server.TThreadPoolServer: Error occurred during processing of
> message.
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException: Peer indicated failure:
> GSS initiate failed
>     at org.apache.thrift.transport.TSaslServerTransport
> $Factory.getTransport(TSaslServerTransport.java:219)
>     at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run
> (TThreadPoolServer.java:189)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1142)
> Agent g2:
> ERROR transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - UNKNOWN_SERVER)]
>     at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge
> (GssKrb5Client.java:211)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Github integration

[Johny Rufus John]

+1. I have seen discussion on other projects contemplating on making this
move.

Thanks,
Rufus

On Sat, Mar 19, 2016 at 1:17 PM, Ashish  wrote:

> +1 for it. Let's try and see if it makes it easy to contribute.
>
> On Sat, Mar 19, 2016 at 12:14 PM, Hari Shreedharan
>  wrote:
> > Hi,
> >
> > I have worked for a while on Spark recently, and like using github for
> > scm. While not the best tool for code reviews, it certainly is better
> > than using patches on jiras. We already get a lot of review requests
> > as Pull Requests anyway. I'd like some community feedback on this.
> >
> > I think we'd need a vote before we can get it done.
> >
> > Thanks!
> > Hari
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

[jira] [Resolved] (FLUME-2735) Kafka Channel needs zookeeper.jar to function without errors

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2735.

Resolution: Duplicate

Closing this as duplicate, as FLUME-2840 has a patch attached, and is also a 
zookeeper dependency issue

> Kafka Channel needs zookeeper.jar to function without errors
> 
>
> Key: FLUME-2735
> URL: https://issues.apache.org/jira/browse/FLUME-2735
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
>
> Currently using kafka channel, gives the following error. Adding the 
> zookeeper.jar (tried both 3.3.6 or 3.4.6) to the classpath helps to get past 
> this error. 
> (With only zkclient:jar pulled in by Kafka, the above error occurs )
> 2015-07-04 08:18:09,201 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO 
> - kafka.utils.Logging$class.info(Logging.scala:68)] 
> [flume_jrufus-MBP.local-1436023089200-e4aacadf], Connecting to zookeeper 
> instance at jrufus-ptest-1.vpc.cloudera.com:2181
> 2015-07-04 08:18:09,203 (SinkRunner-PollingRunner-DefaultSinkProcessor) 
> [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] 
> Unable to deliver event. Exception follows.
> java.lang.IllegalStateException: close() called when transaction is OPEN - 
> you must either commit or rollback first
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:172)
>   at 
> org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
>   at org.apache.flume.sink.LoggerSink.process(LoggerSink.java:105)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2704) Configurable poll delay for spooling directory source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102944#comment-15102944
 ] 

Johny Rufus commented on FLUME-2704:


[~roshan_naik], the changes look good, the patch doesnt apply cleanly, had to 
rebase, will commit this as soon as I run the tests

> Configurable poll delay for spooling directory source
> -
>
> Key: FLUME-2704
> URL: https://issues.apache.org/jira/browse/FLUME-2704
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.6.0, v1.5.2
>Reporter: Somin Mithraa
>Assignee: Somin Mithraa
>Priority: Minor
>  Labels: SpoolDir, pollDelay, sources
> Attachments: FLUME-2704.patch
>
>
> SpoolDir source polls a directory for new files at specific interval. This 
> interval(or poll delay) is currently hardcoded as 500ms.
> 500ms may be too fast for some applications. This JIRA is to make this 
> property configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39035: FLUME-2712: Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus John]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39035/
---

(Updated Oct. 30, 2015, 8:13 p.m.)


Review request for Flume.


Bugs: FLUME-2712
https://issues.apache.org/jira/browse/FLUME-2712


Repository: flume-git


Description
---

Optional channel errors slows down the Source to Main channel event rate. This 
change makes delivery to optional channel asynchronously.


Diffs (updated)
-

  flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 
1cce137 
  
flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelProcessor.java 
0656596 

Diff: https://reviews.apache.org/r/39035/diff/


Testing
---

Tested with optional channel setup to observe the performance improvement


Thanks,

Johny Rufus John

[jira] [Updated] (FLUME-2712) Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2712:
---
Attachment: FLUME-2712-2.patch

[~hshreedharan], Attaching patch with review comments incorporated

> Optional channel errors slows down the Source to Main channel event rate
> 
>
> Key: FLUME-2712
> URL: https://issues.apache.org/jira/browse/FLUME-2712
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2712-1.patch, FLUME-2712-2.patch, 
> FLUME-2712.patch, FLUME-2712.patch
>
>
> When we have a source configured to deliver events to a main channel and an 
> optional channel, and if the delivery to optional channel fails, this 
> significantly slows down the rate at which events are delivered to the main 
> channel by the source.
> We need to evaluate async means of delivering events to the optional channel 
> and isolate the errors happening in optional channel from the delivery to the 
> main channel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2712) Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2712:
---
Attachment: FLUME-2712-1.patch

 Attaching patch with the test case and single thread of execution for optional 
channel

> Optional channel errors slows down the Source to Main channel event rate
> 
>
> Key: FLUME-2712
> URL: https://issues.apache.org/jira/browse/FLUME-2712
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2712-1.patch, FLUME-2712.patch, FLUME-2712.patch
>
>
> When we have a source configured to deliver events to a main channel and an 
> optional channel, and if the delivery to optional channel fails, this 
> significantly slows down the rate at which events are delivered to the main 
> channel by the source.
> We need to evaluate async means of delivering events to the optional channel 
> and isolate the errors happening in optional channel from the delivery to the 
> main channel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39035: FLUME-2712: Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus John]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39035/
---

(Updated Oct. 28, 2015, 12:10 a.m.)


Review request for Flume.


Bugs: FLUME-2712
https://issues.apache.org/jira/browse/FLUME-2712


Repository: flume-git


Description
---

Optional channel errors slows down the Source to Main channel event rate. This 
change makes delivery to optional channel asynchronously.


Diffs (updated)
-

  flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 
1cce137 
  
flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelProcessor.java 
0656596 

Diff: https://reviews.apache.org/r/39035/diff/


Testing
---

Tested with optional channel setup to observe the performance improvement


Thanks,

Johny Rufus John

[jira] [Resolved] (FLUME-2593) ResettableFileInputStream returns negate values from read() method

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2593.

   Resolution: Fixed
 Assignee: Miroslav Holubec
Fix Version/s: v1.7.0

Thanks for the patch [~myr...@gmail.com] !!

> ResettableFileInputStream returns negate values from read() method
> --
>
> Key: FLUME-2593
> URL: https://issues.apache.org/jira/browse/FLUME-2593
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.5.2
>Reporter: Miroslav Holubec
>Assignee: Miroslav Holubec
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: 
> 0001-FLUME-2593-ResettableFileInputStream-returns-negate-.patch, 
> 0001-FLUME-2593-ResettableFileInputStream-returns-negate-.patch
>
>
> the int value returned from the method should be in range 0-255 for a byte 
> read, or -1 when EOF was reached, as described in InputStream class javadoc.
> Fix is to change the line in read() method:
> {noformat}
>  return byteBuf[0];
> {noformat}
> to
> {noformat}
>  return byteBuf[0] & 0xFF;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2593) ResettableFileInputStream returns negate values from read() method

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969429#comment-14969429
 ] 

Johny Rufus commented on FLUME-2593:


[~myroch], Thanks for the patch with test case, running tests now, will commit 
today

> ResettableFileInputStream returns negate values from read() method
> --
>
> Key: FLUME-2593
> URL: https://issues.apache.org/jira/browse/FLUME-2593
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.5.2
>Reporter: Miroslav Holubec
>Priority: Minor
> Attachments: 
> 0001-FLUME-2593-ResettableFileInputStream-returns-negate-.patch, 
> 0001-FLUME-2593-ResettableFileInputStream-returns-negate-.patch
>
>
> the int value returned from the method should be in range 0-255 for a byte 
> read, or -1 when EOF was reached, as described in InputStream class javadoc.
> Fix is to change the line in read() method:
> {noformat}
>  return byteBuf[0];
> {noformat}
> to
> {noformat}
>  return byteBuf[0] & 0xFF;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] Change of Apache Flume PMC Chair

[Johny Rufus]

Congrats Hari !! Wonderful news !!

Regards,
Rufus


On Wed, Oct 21, 2015 at 5:50 PM, Arvind Prabhakar  wrote:

> Dear Flume Users and Developers,
>
> I have had the pleasure of serving as the PMC Chair of Apache Flume since
> its graduation three years ago. I sincerely thank you and the Flume PMC for
> this opportunity. However, I have decided to step down from this
> responsibility due to personal reasons.
>
> I am very happy to announce that on the request of Flume PMC and with the
> approval from the board of directors at The Apache Software Foundation,
> Hari Shreedharan is hereby appointed as the new PMC Chair. I am confident
> that Hari will do everything possible to help further grow the community
> and adoption of Apache Flume.
>
> Please join me in congratulating Hari on his appointment and welcoming him
> to this role.
>
> Regards,
> Arvind Prabhakar
>

[jira] [Commented] (FLUME-2632) High CPU on KafkaSink

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967677#comment-14967677
 ] 

Johny Rufus commented on FLUME-2632:


+1,
Looks like I +1ed this on the review request long time back, will run tests and 
commit 

> High CPU on KafkaSink
> -
>
> Key: FLUME-2632
> URL: https://issues.apache.org/jira/browse/FLUME-2632
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>  Labels: kafka
> Fix For: v1.6.0
>
> Attachments: FLUME-2632-0.patch
>
>
> Reported here: https://github.com/harishreedharan/flume/issues/1
> "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu 
> utilization stay at 100% and never dropped down all the time even at the time 
> the channel is empty.
> I looked into the source code and found that "process" function in KafkaSink 
> always return Status.READY even if no events available in channel. That 
> causes the SinkRunner keep running achieving event from channel and get 
> nothing.
> Do we need to change to return Status.BACKOFF in "process" function in 
> KafkaSink when it notices that there is no events processed in current round? 
> So that the SinkRunner has a chance to take a rest when there is no event in 
> channel. If this proposal feasible, function "testEmptyChannel" in 
> TestKafkaSink also need to be changed. "



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2777) Tail Dir Source leads to duplicate events on rolling the tailed file

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956796#comment-14956796
 ] 

Johny Rufus commented on FLUME-2777:


[~siefried12], you are right, creation time returns last modified time, I have 
been thinking of other ways to solve this w/o using creation time, but not much 
has come up

> Tail Dir Source leads to duplicate events on rolling the tailed file
> 
>
> Key: FLUME-2777
> URL: https://issues.apache.org/jira/browse/FLUME-2777
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: notrack
>Reporter: Johny Rufus
>Assignee: Johny Rufus
> Attachments: FLUME-2777-1.patch, FLUME-2777.patch
>
>
> I have a simple setup, where I write 200 events to logfile1. [TailSrc is on 
> the lookout for logfile* ]
> Then I rename logfile1 to logfile2.
> I create a new logfile1 and write 100 events to it.
> Typically I should see 300 events in my channel. But I see 500 events.
> I was able to trace the duplicates to ReliableTaildirEventReader.java 
> updateFiles(boolean) to the way renamed files are handled , by specifying 
> starting position as 0. [This starting position should be obtained from 
> tf.getPosition()]
> I am attaching a proposed fix, would be great if one of you guys 
> [~iijima_satoshi] / [~hshreedharan]/ [~roshan_naik] can take a look at the 
> fix and validate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2712) Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949257#comment-14949257
 ] 

Johny Rufus commented on FLUME-2712:


Thanks [~hshreedharan] for the catch. I thought we generally dont guarantee 
ordering, but it makes sense not to break the existing level of ordering. 
Submitting to a single threaded executor will guarantee the sequential 
execution of jobs and it works out of an unbounded queue, so let me make that 
change. Will also work on a test case.

> Optional channel errors slows down the Source to Main channel event rate
> 
>
> Key: FLUME-2712
> URL: https://issues.apache.org/jira/browse/FLUME-2712
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2712.patch, FLUME-2712.patch
>
>
> When we have a source configured to deliver events to a main channel and an 
> optional channel, and if the delivery to optional channel fails, this 
> significantly slows down the rate at which events are delivered to the main 
> channel by the source.
> We need to evaluate async means of delivering events to the optional channel 
> and isolate the errors happening in optional channel from the delivery to the 
> main channel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Camel integration/IMAP Source

[Johny Rufus]

We already have a Camel source developed as part of FLUME-2706, you can
take a look at that
https://issues.apache.org/jira/browse/FLUME-2706, am not sure about the
Camel sink.

Thanks,
Rufus

On Thu, Oct 8, 2015 at 5:48 AM, Hendrik Dev  wrote:

> Hi,
>
> is it possible to use Apache Camel Components (for example
> http://camel.apache.org/dropbox.html) as sinks and sources? If not
> would it be helpful to have a CamelSink which redirects to Camel
> Consumers and a CamelSource which uses a CamelProducer? If so i plan
> to start such a development. I am especially interested in an IMAP
> Source (https://issues.apache.org/jira/browse/FLUME-2000) and either i
> plan to develop a dedicated Source for that or use the above mentioned
> "Camel Bridge" for that.
>
> Thanks
> Hendrik
>
> (I am an Apache Committer on Johnzon (incubator) and Geronimo and
> would like to get active on Flume)
>
> --
> Hendrik Saly (salyh, hendrikdev22)
> @hendrikdev22
> PGP: 0x22D7F6EC
>

Re: Review Request 39035: FLUME-2712: Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus John]

> On Oct. 6, 2015, 4:11 p.m., Jarek Cecho wrote:
> > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java, 
> > line 86
> > <https://reviews.apache.org/r/39035/diff/1/?file=1091484#file1091484line86>
> >
> > Shouldn't we document the new configuration option?

Yes, we have to, since this option is going to be documented in every source, I 
thought we can do that as part of separate Jira


- Johny Rufus


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39035/#review101651
-------


On Oct. 6, 2015, 1:30 a.m., Johny Rufus John wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39035/
> ---
> 
> (Updated Oct. 6, 2015, 1:30 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2712
> https://issues.apache.org/jira/browse/FLUME-2712
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Optional channel errors slows down the Source to Main channel event rate. 
> This change makes delivery to optional channel asynchronously.
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 
> 1cce137 
> 
> Diff: https://reviews.apache.org/r/39035/diff/
> 
> 
> Testing
> ---
> 
> Tested with optional channel setup to observe the performance improvement
> 
> 
> Thanks,
> 
> Johny Rufus John
> 
>

[jira] [Updated] (FLUME-2712) Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2712:
---
Attachment: FLUME-2712.patch

> Optional channel errors slows down the Source to Main channel event rate
> 
>
> Key: FLUME-2712
> URL: https://issues.apache.org/jira/browse/FLUME-2712
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2712.patch, FLUME-2712.patch
>
>
> When we have a source configured to deliver events to a main channel and an 
> optional channel, and if the delivery to optional channel fails, this 
> significantly slows down the rate at which events are delivered to the main 
> channel by the source.
> We need to evaluate async means of delivering events to the optional channel 
> and isolate the errors happening in optional channel from the delivery to the 
> main channel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2712) Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2712:
---
Attachment: FLUME-2712.patch

Attached patch with the changes to deliver to optional channel asynchronously
- review board link - 
https://reviews.apache.org/r/39035/

> Optional channel errors slows down the Source to Main channel event rate
> 
>
> Key: FLUME-2712
> URL: https://issues.apache.org/jira/browse/FLUME-2712
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2712.patch
>
>
> When we have a source configured to deliver events to a main channel and an 
> optional channel, and if the delivery to optional channel fails, this 
> significantly slows down the rate at which events are delivered to the main 
> channel by the source.
> We need to evaluate async means of delivering events to the optional channel 
> and isolate the errors happening in optional channel from the delivery to the 
> main channel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 39035: FLUME-2712: Optional channel errors slows down the Source to Main channel event rate

[Johny Rufus John]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39035/
---

Review request for Flume.


Bugs: FLUME-2712
https://issues.apache.org/jira/browse/FLUME-2712


Repository: flume-git


Description
---

Optional channel errors slows down the Source to Main channel event rate. This 
change makes delivery to optional channel asynchronously.


Diffs
-

  flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 
1cce137 

Diff: https://reviews.apache.org/r/39035/diff/


Testing
---

Tested with optional channel setup to observe the performance improvement


Thanks,

Johny Rufus John

[jira] [Updated] (FLUME-2773) TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2773:
---
Fix Version/s: v1.7.0

> TailDirSource throws FileNotFound Exception if ~/.flume directory is not 
> created already
> 
>
> Key: FLUME-2773
> URL: https://issues.apache.org/jira/browse/FLUME-2773
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.7.0
>    Reporter: Johny Rufus
>Assignee: Johny Rufus
> Fix For: v1.7.0
>
> Attachments: FLUME-2773.patch
>
>
> If we leave the positionFile parameter to default, 
> then the following exception is thrown when ~/.flume is not present
> [We should take care of creating the default directory if not present ]
> 2015-08-27 09:44:30,551 (positionWriter) [ERROR - 
> org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:312)]
>  Failed writing positionFile
> java.io.FileNotFoundException: /Users/jrufus/.flume/taildir_position.json (No 
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at java.io.FileWriter.(FileWriter.java:90)
> at 
> org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:306)
> at 
> org.apache.flume.source.taildir.TaildirSource.access$600(TaildirSource.java:56)
> at 
> org.apache.flume.source.taildir.TaildirSource$PositionWriterRunnable.run(TaildirSource.java:298)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2437) S3 Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942500#comment-14942500
 ] 

Johny Rufus commented on FLUME-2437:


[~ashishpaliwal], rebase done (phew !) also ported/made changes to 
GenericResettableInputStream to take care of the fix from FLUME-2215 ucs-4 
support 

Will attach the final patch once tests pass + basic end-end integration 
scenarios pass.

> S3 Source
> -
>
> Key: FLUME-2437
> URL: https://issues.apache.org/jira/browse/FLUME-2437
> Project: Flume
>  Issue Type: New Feature
>Reporter: Jonathan Natkins
>    Assignee: Johny Rufus
> Attachments: FLUME-2437-2.patch, FLUME-2437.patch
>
>
> There have been multiple requests on the mailing list for an S3 source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2437) S3 Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940414#comment-14940414
 ] 

Johny Rufus commented on FLUME-2437:


Sure [~ashishpaliwal], I think I addressed all the review comments you had 
mentioned from last time.

> S3 Source
> -
>
> Key: FLUME-2437
> URL: https://issues.apache.org/jira/browse/FLUME-2437
> Project: Flume
>  Issue Type: New Feature
>Reporter: Jonathan Natkins
>    Assignee: Johny Rufus
> Attachments: FLUME-2437-2.patch, FLUME-2437.patch
>
>
> There have been multiple requests on the mailing list for an S3 source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-1670) AvroClient/Agent should return with valid error code when it terminates in error state

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-1670:
---
Attachment: FLUME-1670-1.patch

Attaching rebased/adjusted patch based on [~paulchavez]'s patch

> AvroClient/Agent should return with valid error code when it terminates in 
> error state
> --
>
> Key: FLUME-1670
> URL: https://issues.apache.org/jira/browse/FLUME-1670
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.4.0
>Reporter: Hari Shreedharan
> Attachments: FLUME-1670-1.patch, FLUME-1670.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2095) JMS source with TIBCO

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901223#comment-14901223
 ] 

Johny Rufus commented on FLUME-2095:


+1, Looks good, running tests and committing 

> JMS source with TIBCO
> -
>
> Key: FLUME-2095
> URL: https://issues.apache.org/jira/browse/FLUME-2095
> Project: Flume
>  Issue Type: Question
>  Components: Sinks+Sources
>Affects Versions: v1.5.0
> Environment: Windows 7
>Reporter: Bhaskar Reddy
>Priority: Critical
>  Labels: easyfix, features, jms
> Attachments: FLUME-2095-0.patch, FLUME-2095-1.patch, 
> FLUME-2095-2.patch
>
>
> Hi,
> I was trying to use the JMS source to work with TIBCO, but I am encountering 
> the exception below,
> org.apache.flume.FlumeException: Could not lookup ConnectionFactory
>   at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:222)
>   at 
> org.apache.flume.source.BasicSourceSemantics.configure(BasicSourceSemantics.java:65)
>   at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>   at 
> org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
>   at 
> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
>   at 
> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>   at 
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>   at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.naming.NameNotFoundException: Name not found: 
> 'com.tibco.tibjms.TibjmsQueueConnectionFactory'
>   at com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:715)
>   at com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:491)
>   at javax.naming.InitialContext.lookup(InitialContext.java:392)
>   at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:219)
>   ... 14 more
> Please find my configuration below,
> a1.sources.r1.type = jms
> a1.sources.r1.channels = c1
> a1.sources.r1.initialContextFactory = 
> com.tibco.tibjms.naming.TibjmsInitialContextFactory
> a1.sources.r1.connectionFactory = 
> com.tibco.tibjms.TibjmsQueueConnectionFactory
> a1.sources.r1.providerURL = tibjmsnaming://localhost:7222
> a1.sources.r1.destinationName = sample
> a1.sources.r1.destinationType = QUEUE
> I tried changing the configuration below,
> a1.sources.r1.type = jms
> a1.sources.r1.channels = c1
> a1.sources.r1.initialContextFactory = 
> com.tibco.tibjms.naming.TibjmsInitialContextFactory
> a1.sources.r1.providerURL = tcp://localhost:7222
> a1.sources.r1.destinationName = sample
> a1.sources.r1.destinationType = QUEUE
> Thanks,
> Bhaskar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2758) Shield Support for Elasticsearch Sink

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900748#comment-14900748
 ] 

Johny Rufus commented on FLUME-2758:


[~crondaemon], Added you to the contributor's list, from now on you can assign 
any issue to yourself

> Shield Support for Elasticsearch Sink
> -
>
> Key: FLUME-2758
> URL: https://issues.apache.org/jira/browse/FLUME-2758
> Project: Flume
>  Issue Type: New Feature
>Affects Versions: v1.6.0
>Reporter: Jared Ramsey
>Assignee: Dario Lombardo
>
> The existing Elasticsearch sink does not support Elasticsearch clusters that 
> have been secured with Shield.  (See https://www.elastic.co/products/shield)  
> All that would be required to add this support is allowing users to configure 
> the username and password on the sink in their flume.conf and then setting 
> the username and password on the HTTP header like so:
> UsernamePasswordCredentials creds = new 
> UsernamePasswordCredentials("username", "password");
> HttpPost httpRequest = new HttpPost(url);
> httpRequest.addHeader(new BasicScheme().authenticate(creds, httpRequest));
> I've already hard-coded a quick fix for myself that works, but it would be 
> awesome to see this in a future release and/or patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2758) Shield Support for Elasticsearch Sink

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2758:
---
Assignee: Dario Lombardo

> Shield Support for Elasticsearch Sink
> -
>
> Key: FLUME-2758
> URL: https://issues.apache.org/jira/browse/FLUME-2758
> Project: Flume
>  Issue Type: New Feature
>Affects Versions: v1.6.0
>Reporter: Jared Ramsey
>Assignee: Dario Lombardo
>
> The existing Elasticsearch sink does not support Elasticsearch clusters that 
> have been secured with Shield.  (See https://www.elastic.co/products/shield)  
> All that would be required to add this support is allowing users to configure 
> the username and password on the sink in their flume.conf and then setting 
> the username and password on the HTTP header like so:
> UsernamePasswordCredentials creds = new 
> UsernamePasswordCredentials("username", "password");
> HttpPost httpRequest = new HttpPost(url);
> httpRequest.addHeader(new BasicScheme().authenticate(creds, httpRequest));
> I've already hard-coded a quick fix for myself that works, but it would be 
> awesome to see this in a future release and/or patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2777) Tail Dir Source leads to duplicate events on rolling the tailed file

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2777:
---
Attachment: FLUME-2777-1.patch

[~iijima_satoshi], I have attached a patch that addresses your concern 
regarding Inodes being reused in which case, we need to read from beginning.
I have used a similar logic as before, but in addition, I have handled 2 cases, 
when the file names do not match
1) if its a renamed file, in which case we read from last position from Tail 
File
2) its the case of deleted/truncated file and inode being reused in which case, 
we read the file from the beginning
(the difference between the two cases above is identified by the Creation Time)

> Tail Dir Source leads to duplicate events on rolling the tailed file
> 
>
> Key: FLUME-2777
> URL: https://issues.apache.org/jira/browse/FLUME-2777
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.7
>    Reporter: Johny Rufus
>Assignee: Johny Rufus
> Attachments: FLUME-2777-1.patch, FLUME-2777.patch
>
>
> I have a simple setup, where I write 200 events to logfile1. [TailSrc is on 
> the lookout for logfile* ]
> Then I rename logfile1 to logfile2.
> I create a new logfile1 and write 100 events to it.
> Typically I should see 300 events in my channel. But I see 500 events.
> I was able to trace the duplicates to ReliableTaildirEventReader.java 
> updateFiles(boolean) to the way renamed files are handled , by specifying 
> starting position as 0. [This starting position should be obtained from 
> tf.getPosition()]
> I am attaching a proposed fix, would be great if one of you guys 
> [~iijima_satoshi] / [~hshreedharan]/ [~roshan_naik] can take a look at the 
> fix and validate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2783) Update Website Team page with new Committer's

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2783.

Resolution: Fixed
  Assignee: Johny Rufus

Staged and published 

> Update Website Team page with new Committer's
> -
>
> Key: FLUME-2783
> URL: https://issues.apache.org/jira/browse/FLUME-2783
> Project: Flume
>  Issue Type: Documentation
>  Components: Docs
>Reporter: Ashish Paliwal
>Assignee: Johny Rufus
>Priority: Minor
> Attachments: FLUME-2783.patch
>
>
> Team page on website need to add 2 newly voted committers
> Ashish Paliwal
> Johny Rufus



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2783) Update Website Team page with new Committer's

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2783:
---
Attachment: FLUME-2783.patch

[~ashishpaliwal], can you verify the changes ? 


> Update Website Team page with new Committer's
> -
>
> Key: FLUME-2783
> URL: https://issues.apache.org/jira/browse/FLUME-2783
> Project: Flume
>  Issue Type: Documentation
>  Components: Docs
>Reporter: Ashish Paliwal
>Priority: Minor
> Attachments: FLUME-2783.patch
>
>
> Team page on website need to add 2 newly voted committers
> Ashish Paliwal
> Johny Rufus



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746632#comment-14746632
 ] 

Johny Rufus commented on FLUME-2672:


+1, Running tests and committing

> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Assignee: Rigo MacTaggart
>Priority: Trivial
>  Labels: easyfix
> Attachments: FLUME-2672-with-test-git-diff.patch, 
> FLUME-2672-with-test.patch, FLUME-2672.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() 
> because it expects MonitoredCounterGroup.counterMap to contain key 
> "source.kafka.empty.count". A patch is included which adds this key to the 
> ATTRIBUTES string array, which is used to pre-populate keys with an initial 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flume events are truncated beyond 2048 characters

[Johny Rufus]

I believe there was a recent enquiry related to truncation beyond 2048
characters.
If you are using Line Deserializer in your source, the default maximum is
2048. You can change the property as per this -
https://flume.apache.org/FlumeUserGuide.html#line

Thanks,
Rufus

On Thu, Sep 10, 2015 at 10:12 AM, Gonzalo Herreros 
wrote:

> I'm sending much bigger messages without issues.
> Do you know at which point it gets truncated: source, channel or sink?
>
> Regards,
> Gonzalo
>
> On 10 September 2015 at 18:07, Robin Jain  wrote:
>
> > Hi all,
> >
> > I am facing issue with events truncation beyond 2048 characters.
> >
> > Is there any way to handle such scenario .
> >
> > Sincerely,
> >
> > Robin Jain
> >
>

[jira] [Updated] (FLUME-2777) Tail Dir Source leads to duplicate events on rolling the tailed file

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2777:
---
Attachment: FLUME-2777.patch

> Tail Dir Source leads to duplicate events on rolling the tailed file
> 
>
> Key: FLUME-2777
> URL: https://issues.apache.org/jira/browse/FLUME-2777
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.7
>    Reporter: Johny Rufus
>Assignee: Johny Rufus
> Attachments: FLUME-2777.patch
>
>
> I have a simple setup, where I write 200 events to logfile1. [TailSrc is on 
> the lookout for logfile* ]
> Then I rename logfile1 to logfile2.
> I create a new logfile1 and write 100 events to it.
> Typically I should see 300 events in my channel. But I see 500 events.
> I was able to trace the duplicates to ReliableTaildirEventReader.java 
> updateFiles(boolean) to the way renamed files are handled , by specifying 
> starting position as 0. [This starting position should be obtained from 
> tf.getPosition()]
> I am attaching a proposed fix, would be great if one of you guys 
> [~iijima_satoshi] / [~hshreedharan]/ [~roshan_naik] can take a look at the 
> fix and validate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2777) Tail Dir Source leads to duplicate events on rolling the tailed file

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2777:
--

 Summary: Tail Dir Source leads to duplicate events on rolling the 
tailed file
 Key: FLUME-2777
 URL: https://issues.apache.org/jira/browse/FLUME-2777
 Project: Flume
  Issue Type: Bug
  Components: Sinks+Sources
Affects Versions: 1.7
Reporter: Johny Rufus
Assignee: Johny Rufus


I have a simple setup, where I write 200 events to logfile1. [TailSrc is on the 
lookout for logfile* ]
Then I rename logfile1 to logfile2.
I create a new logfile1 and write 100 events to it.

Typically I should see 300 events in my channel. But I see 500 events.
I was able to trace the duplicates to ReliableTaildirEventReader.java 
updateFiles(boolean) to the way renamed files are handled , by specifying 
starting position as 0. [This starting position should be obtained from 
tf.getPosition()]

I am attaching a proposed fix, would be great if one of you guys 
[~iijima_satoshi] / [~hshreedharan]/ [~roshan_naik] can take a look at the fix 
and validate the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2776) 1.6 Download link not actually Flume

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720345#comment-14720345
 ] 

Johny Rufus commented on FLUME-2776:


It looks like thats a direct link to the artifact, but its actually a link to 
the Downloads page that lists different mirrors. If you had downloaded the 
initial link, you must have the html downloaded.

> 1.6 Download link not actually Flume
> 
>
> Key: FLUME-2776
> URL: https://issues.apache.org/jira/browse/FLUME-2776
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.6
> Environment: Cygwin running on Windows 7 
>Reporter: Bryan Glazer
>Priority: Critical
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This link on the Flume download page: 
> http://www.apache.org/dyn/closer.cgi/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
> downloads a 4.7kb file that doesn't match the md5sum given and obviously is 
> not a valid Flume binary. 
> I'm unsure if this is a simple mistake or a malicious file. 
> Going to the main distribution page (http://www.apache.org/dist/flume/1.6.0/) 
> gives a valid version of flume. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2776) 1.6 Download link not actually Flume

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14719982#comment-14719982
 ] 

Johny Rufus commented on FLUME-2776:


Can you go to the page you mentioned and select the Download from one of the 
links/mirrors there.

> 1.6 Download link not actually Flume
> 
>
> Key: FLUME-2776
> URL: https://issues.apache.org/jira/browse/FLUME-2776
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.6
> Environment: Cygwin running on Windows 7 
>Reporter: Bryan Glazer
>Priority: Critical
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This link on the Flume download page: 
> http://www.apache.org/dyn/closer.cgi/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
> downloads a 4.7kb file that doesn't match the md5sum given and obviously is 
> not a valid Flume binary. 
> I'm unsure if this is a simple mistake or a malicious file. 
> Going to the main distribution page (http://www.apache.org/dist/flume/1.6.0/) 
> gives a valid version of flume. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2773) TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2773:
---
Attachment: FLUME-2773.patch

Attached patch, that makes sure to create any missing parent directories in the 
positionFile path (either default or user given input)
This ensures that position files are always created [Currently if default 
postionPath is assumed and .flume is not there, then the position checkpointing 
does not happen and user is not made aware of this fact, unless the user looks 
in the log file for the exception]

> TailDirSource throws FileNotFound Exception if ~/.flume directory is not 
> created already
> 
>
> Key: FLUME-2773
> URL: https://issues.apache.org/jira/browse/FLUME-2773
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.7.0
>    Reporter: Johny Rufus
>Assignee: Johny Rufus
> Attachments: FLUME-2773.patch
>
>
> If we leave the positionFile parameter to default, 
> then the following exception is thrown when ~/.flume is not present
> [We should take care of creating the default directory if not present ]
> 2015-08-27 09:44:30,551 (positionWriter) [ERROR - 
> org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:312)]
>  Failed writing positionFile
> java.io.FileNotFoundException: /Users/jrufus/.flume/taildir_position.json (No 
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at java.io.FileWriter.(FileWriter.java:90)
> at 
> org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:306)
> at 
> org.apache.flume.source.taildir.TaildirSource.access$600(TaildirSource.java:56)
> at 
> org.apache.flume.source.taildir.TaildirSource$PositionWriterRunnable.run(TaildirSource.java:298)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2773) TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2773:
--

 Summary: TailDirSource throws FileNotFound Exception if ~/.flume 
directory is not created already
 Key: FLUME-2773
 URL: https://issues.apache.org/jira/browse/FLUME-2773
 Project: Flume
  Issue Type: Bug
Affects Versions: v1.7.0
Reporter: Johny Rufus
Assignee: Johny Rufus


If we leave the positionFile parameter to default, 
then the following exception is thrown when ~/.flume is not present
[We should take care of creating the default directory if not present ]

2015-08-27 09:44:30,551 (positionWriter) [ERROR - 
org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:312)]
 Failed writing positionFile
java.io.FileNotFoundException: /Users/jrufus/.flume/taildir_position.json (No 
such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:221)
at java.io.FileOutputStream.(FileOutputStream.java:171)
at java.io.FileWriter.(FileWriter.java:90)
at 
org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:306)
at 
org.apache.flume.source.taildir.TaildirSource.access$600(TaildirSource.java:56)
at 
org.apache.flume.source.taildir.TaildirSource$PositionWriterRunnable.run(TaildirSource.java:298)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2433) Add kerberos support for Hive sink

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708496#comment-14708496
 ] 

Johny Rufus commented on FLUME-2433:


Sure [~roshan_naik], flume-ng-auth module has all the kerberos authentication 
related code. We probably should leverage the authentication stuff from there, 
currently we have a model where all the flume components that need 
authentication (HDFS sink, HBase sink , Thrist src/sink and Dataset sink) are 
required to use the same credentials [More like authentication creds for the 
entire Flume agent as a whole]

> Add kerberos support for Hive sink
> --
>
> Key: FLUME-2433
> URL: https://issues.apache.org/jira/browse/FLUME-2433
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: HiveSink, Kerberos,
> Attachments: FLUME-2433.patch
>
>
> Add kerberos authentication support for Hive sink
> FYI: The HCatalog API support for Kerberos is not available in hive 0.13.1 
> this should be available in the next hive release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2706) Camel source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705465#comment-14705465
 ] 

Johny Rufus commented on FLUME-2706:


[~dgreco], can you diff from trunk to the changes in your branch, looks like 
the diff posted refers in the reverse way

> Camel source
> 
>
> Key: FLUME-2706
> URL: https://issues.apache.org/jira/browse/FLUME-2706
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: David Greco
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: flume-2706.patch
>
>
> This component can start Camel routes, in this way provides a very powerful 
> mechanism for ingesting data from virtually any source supported by Camel 
> http://camel.apache.org.
> This source can be configured either with a Camel URI or with an XML file 
> containing route specifications.
> The configuration is very simple, let's show a couple of examples:
> 1. Configuration by an URI:
> tier1.sources = source1
> tier1.channels = channel1
> tier1.sinks  = sink1
> tier1.sources.source1.type = org.apache.flume.source.camel.CamelSource
> tier1.sources.source1.sourceURI  = 
> twitter://streaming/sample?type=event&consumerKey=&consumerSecret=&accessToken=
> any URI supported by Camel components is valid. For Twitter, see 
> [here](http://camel.apache.org/twitter.html).
> 2. Configuration by an xml file:
> tier1.sources = source1
> tier1.channels  = channel1
> tier1.sinks = sink1
> tier1.sources.source1.type = org.apache.flume.source.camel.CamelSource
> tier1.sources.source1.routesFile = conf/routes.xml
> where the routes.xml can contain something like:
> http://camel.apache.org/schema/spring";>
> 
>  uri="twitter://streaming/sample?type=event&consumerKey=&consumerSecret=&accessToken="/>
> 
> 
> 
> any route that wants to send data to flume must have as a target endpoint an 
> URI with the following format: direct-vm://, so in 
> our case the CamelSource's name is source1 and consequently the endpoint URI 
> is: direct-vm://source1 as shown in the XML snippet above.
> The sourceURI property takes always precedence on the routesFile property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2763) flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2763.

   Resolution: Fixed
Fix Version/s: v1.7.0

Thanks [~philip] and [~neerjakhattar] for the patch !!

> flume_env script should handle jvm parameters like -javaagent -agentpath 
> -agentlib
> --
>
> Key: FLUME-2763
> URL: https://issues.apache.org/jira/browse/FLUME-2763
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>Assignee: Neerja Khattar
> Fix For: v1.7.0
>
> Attachments: FLUME-2763.patch
>
>
> flume_env script should handle jvm parameters like -javaagent -agentpath and 
> -agentlib when given as command line arguments [Currently only -X -D are 
> handled as java properties]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLUME-2763) flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705048#comment-14705048
 ] 

Johny Rufus edited comment on FLUME-2763 at 8/20/15 2:54 PM:
-

+1, Thanks [~philip] for the patch and [~neerjakhattar] for testing the patch 
in different envrionments. Committing this.


was (Author: jrufus):
Thanks [~philip] for the patch and [~neerjakhattar] for testing the patch in 
different envrionments. Committing this.

> flume_env script should handle jvm parameters like -javaagent -agentpath 
> -agentlib
> --
>
> Key: FLUME-2763
> URL: https://issues.apache.org/jira/browse/FLUME-2763
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>Assignee: Neerja Khattar
> Attachments: FLUME-2763.patch
>
>
> flume_env script should handle jvm parameters like -javaagent -agentpath and 
> -agentlib when given as command line arguments [Currently only -X -D are 
> handled as java properties]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2763) flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705048#comment-14705048
 ] 

Johny Rufus commented on FLUME-2763:


Thanks [~philip] for the patch and [~neerjakhattar] for testing the patch in 
different envrionments. Committing this.

> flume_env script should handle jvm parameters like -javaagent -agentpath 
> -agentlib
> --
>
> Key: FLUME-2763
> URL: https://issues.apache.org/jira/browse/FLUME-2763
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>Assignee: Neerja Khattar
> Attachments: FLUME-2763.patch
>
>
> flume_env script should handle jvm parameters like -javaagent -agentpath and 
> -agentlib when given as command line arguments [Currently only -X -D are 
> handled as java properties]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2437) S3 Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704948#comment-14704948
 ] 

Johny Rufus commented on FLUME-2437:


1.7 should definitely have this. Will try to get this in soon.

> S3 Source
> -
>
> Key: FLUME-2437
> URL: https://issues.apache.org/jira/browse/FLUME-2437
> Project: Flume
>  Issue Type: New Feature
>Reporter: Jonathan Natkins
>Assignee: Ashish Paliwal
> Attachments: FLUME-2437-2.patch, FLUME-2437.patch
>
>
> There have been multiple requests on the mailing list for an S3 source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2498) Implement Taildir Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698769#comment-14698769
 ] 

Johny Rufus commented on FLUME-2498:


+1 for the changes related to ConsumeOrder


> Implement Taildir Source
> 
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, 
> FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time 
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files 
> rotate.
> * It periodically writes the last read position of each file in a position 
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from 
> the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2498) Implement Taildir Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698094#comment-14698094
 ] 

Johny Rufus commented on FLUME-2498:


[~roshan_naik], verified changes from patch v4  are there in the latest patch.
Will try to look at the consume order changes in some time.

> Implement Taildir Source
> 
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, 
> FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time 
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files 
> rotate.
> * It periodically writes the last read position of each file in a position 
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from 
> the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2498) Implement Taildir Source

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2498:
---
Attachment: FLUME-2498-4.patch

Attached FLUME-2498-4.patch that includes 
1) handling of \n and \r\n along with a test case to test the line boundaries
2) A couple of doc additions that mention that the binary files are not 
supported and this source does not rename or delete or do any modifications to 
the file being tailed [Both doc changes are as per the comments in this 
discussion thread]

> Implement Taildir Source
> 
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, 
> FLUME-2498-4.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time 
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files 
> rotate.
> * It periodically writes the last read position of each file in a position 
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from 
> the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2498) Implement Taildir Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694262#comment-14694262
 ] 

Johny Rufus commented on FLUME-2498:


Sure, working on it, let me attach the patch with extra test case, once done

> Implement Taildir Source
> 
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time 
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files 
> rotate.
> * It periodically writes the last read position of each file in a position 
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from 
> the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2498) Implement Taildir Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693975#comment-14693975
 ] 

Johny Rufus commented on FLUME-2498:


Hi [~roshan_naik], Using the System.lineSeparator() is going to return the 
current system's line separator which may not be the same case with the file 
being processed. 
So typically we should 
1) figure out the end of a line using '\n' (should work for both unix and 
windows)
2) and remove '\n' or '\r\n' in the end  depending upon which one is present 
(should work for both unix and windows)

Let me know if this sounds good.

> Implement Taildir Source
> 
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time 
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files 
> rotate.
> * It periodically writes the last read position of each file in a position 
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from 
> the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2498) Implement Taildir Source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693660#comment-14693660
 ] 

Johny Rufus commented on FLUME-2498:


[~roshan_naik], let me look at the new line issue.

> Implement Taildir Source
> 
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time 
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files 
> rotate.
> * It periodically writes the last read position of each file in a position 
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from 
> the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2763) flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2763:
---
Assignee: Neerja Khattar

> flume_env script should handle jvm parameters like -javaagent -agentpath 
> -agentlib
> --
>
> Key: FLUME-2763
> URL: https://issues.apache.org/jira/browse/FLUME-2763
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>Assignee: Neerja Khattar
>
> flume_env script should handle jvm parameters like -javaagent -agentpath and 
> -agentlib when given as command line arguments [Currently only -X -D are 
> handled as java properties]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2763) flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2763:
--

 Summary: flume_env script should handle jvm parameters like 
-javaagent -agentpath -agentlib
 Key: FLUME-2763
 URL: https://issues.apache.org/jira/browse/FLUME-2763
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus


flume_env script should handle jvm parameters like -javaagent -agentpath and 
-agentlib when given as command line arguments [Currently only -X -D are 
handled as java properties]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2753) Error when specifying empty replace string in Search and Replace Interceptor

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660578#comment-14660578
 ] 

Johny Rufus commented on FLUME-2753:


[~ashishpaliwal], if the patch looks fine, can you look at committing this 

> Error when specifying empty replace string in Search and Replace Interceptor
> 
>
> Key: FLUME-2753
> URL: https://issues.apache.org/jira/browse/FLUME-2753
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2753-1.patch, FLUME-2753.patch
>
>
> The user guide specifies that the following configuration
> a1.sources.avroSrc.interceptors.search-replace.replaceString =
> will replace the search pattern with an empty string, but the following error 
> is observed and there is no way to specify an empty replace string
> java.lang.NullPointerException: Must supply a replacement string 
> replaceString (empty is ok)
> at 
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at 
> org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108)
> at 
> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110)
> We need to modify the SearchAndReplaceInterceptor to accept null values for 
> replacement string and use empty string in that case



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2485) Thrift Source tests fail on Oracle JDK 8

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2485.

   Resolution: Fixed
Fix Version/s: v1.7.0

Committed. Thanks Miroslav Holubec !!

> Thrift Source tests fail on Oracle JDK 8
> 
>
> Key: FLUME-2485
> URL: https://issues.apache.org/jira/browse/FLUME-2485
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Santiago M. Mola
>Assignee: Miroslav Holubec
>  Labels: jdk8
> Fix For: v1.7.0
>
> Attachments: 
> 0001-FLUME-2485-too-fast-processing-leads-to-failed-jUnit.patch, 
> FLUME-2485.patch
>
>
> Thrift Source tests fail on Oracle JDK 8:
> https://travis-ci.org/Stratio/flume/jobs/36817396#L6245
> testAppendBatch(org.apache.flume.source.TestThriftSource)  Time elapsed: 6083 
> sec  <<< FAILURE!
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:92)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at org.junit.Assert.assertTrue(Assert.java:54)
>   at 
> org.apache.flume.source.TestThriftSource.testAppendBatch(TestThriftSource.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2753) Error when specifying empty replace string in Search and Replace Interceptor

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2753:
---
Attachment: FLUME-2753-1.patch

Thanks [~ashishpaliwal] for looking into this, attached patch with test case

> Error when specifying empty replace string in Search and Replace Interceptor
> 
>
> Key: FLUME-2753
> URL: https://issues.apache.org/jira/browse/FLUME-2753
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2753-1.patch, FLUME-2753.patch
>
>
> The user guide specifies that the following configuration
> a1.sources.avroSrc.interceptors.search-replace.replaceString =
> will replace the search pattern with an empty string, but the following error 
> is observed and there is no way to specify an empty replace string
> java.lang.NullPointerException: Must supply a replacement string 
> replaceString (empty is ok)
> at 
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at 
> org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108)
> at 
> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110)
> We need to modify the SearchAndReplaceInterceptor to accept null values for 
> replacement string and use empty string in that case



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2485) Thrift Source tests fail on Oracle JDK 8

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2485:
---
Attachment: FLUME-2485.patch

> Thrift Source tests fail on Oracle JDK 8
> 
>
> Key: FLUME-2485
> URL: https://issues.apache.org/jira/browse/FLUME-2485
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Santiago M. Mola
>Assignee: Miroslav Holubec
>  Labels: jdk8
> Attachments: 
> 0001-FLUME-2485-too-fast-processing-leads-to-failed-jUnit.patch, 
> FLUME-2485.patch
>
>
> Thrift Source tests fail on Oracle JDK 8:
> https://travis-ci.org/Stratio/flume/jobs/36817396#L6245
> testAppendBatch(org.apache.flume.source.TestThriftSource)  Time elapsed: 6083 
> sec  <<< FAILURE!
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:92)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at org.junit.Assert.assertTrue(Assert.java:54)
>   at 
> org.apache.flume.source.TestThriftSource.testAppendBatch(TestThriftSource.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2485) Thrift Source tests fail on Oracle JDK 8

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2485:
---
Assignee: Miroslav Holubec

> Thrift Source tests fail on Oracle JDK 8
> 
>
> Key: FLUME-2485
> URL: https://issues.apache.org/jira/browse/FLUME-2485
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Santiago M. Mola
>Assignee: Miroslav Holubec
>  Labels: jdk8
> Attachments: 
> 0001-FLUME-2485-too-fast-processing-leads-to-failed-jUnit.patch
>
>
> Thrift Source tests fail on Oracle JDK 8:
> https://travis-ci.org/Stratio/flume/jobs/36817396#L6245
> testAppendBatch(org.apache.flume.source.TestThriftSource)  Time elapsed: 6083 
> sec  <<< FAILURE!
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:92)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at org.junit.Assert.assertTrue(Assert.java:54)
>   at 
> org.apache.flume.source.TestThriftSource.testAppendBatch(TestThriftSource.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2485) Thrift Source tests fail on Oracle JDK 8

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654036#comment-14654036
 ] 

Johny Rufus commented on FLUME-2485:


+1, Running into this issue a lot, since we have not heard back from 
[~myr...@gmail.com], am renaming/rebasing (if needed) the patch to our 
convention and committing this.

> Thrift Source tests fail on Oracle JDK 8
> 
>
> Key: FLUME-2485
> URL: https://issues.apache.org/jira/browse/FLUME-2485
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Santiago M. Mola
>  Labels: jdk8
> Attachments: 
> 0001-FLUME-2485-too-fast-processing-leads-to-failed-jUnit.patch
>
>
> Thrift Source tests fail on Oracle JDK 8:
> https://travis-ci.org/Stratio/flume/jobs/36817396#L6245
> testAppendBatch(org.apache.flume.source.TestThriftSource)  Time elapsed: 6083 
> sec  <<< FAILURE!
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:92)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at org.junit.Assert.assertTrue(Assert.java:54)
>   at 
> org.apache.flume.source.TestThriftSource.testAppendBatch(TestThriftSource.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2753) Error when specifying empty replace string in Search and Replace Interceptor

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2753:
---
Attachment: FLUME-2753.patch

This change will assign Empty string ""  as the replacement String when the 
configuration for replacement String is absent or is configured as 
a1.sources.avroSrc.interceptors.search-replace.replaceString =

> Error when specifying empty replace string in Search and Replace Interceptor
> 
>
> Key: FLUME-2753
> URL: https://issues.apache.org/jira/browse/FLUME-2753
> Project: Flume
>  Issue Type: Bug
>Reporter: Johny Rufus
>Assignee: Johny Rufus
> Attachments: FLUME-2753.patch
>
>
> The user guide specifies that the following configuration
> a1.sources.avroSrc.interceptors.search-replace.replaceString =
> will replace the search pattern with an empty string, but the following error 
> is observed and there is no way to specify an empty replace string
> java.lang.NullPointerException: Must supply a replacement string 
> replaceString (empty is ok)
> at 
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at 
> org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108)
> at 
> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110)
> We need to modify the SearchAndReplaceInterceptor to accept null values for 
> replacement string and use empty string in that case



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2753) Error when specifying empty replace string in Search and Replace Interceptor

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2753:
--

 Summary: Error when specifying empty replace string in Search and 
Replace Interceptor
 Key: FLUME-2753
 URL: https://issues.apache.org/jira/browse/FLUME-2753
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


The user guide specifies that the following configuration

a1.sources.avroSrc.interceptors.search-replace.replaceString =

will replace the search pattern with an empty string, but the following error 
is observed and there is no way to specify an empty replace string

java.lang.NullPointerException: Must supply a replacement string replaceString 
(empty is ok)
at 
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at 
org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108)
at 
org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110)

We need to modify the SearchAndReplaceInterceptor to accept null values for 
replacement string and use empty string in that case



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2734) Kafka Channel timeout property is overridden by default value

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2734:
---
Attachment: FLUME-2734.patch

Attaching patch which enables kafka.consumer.timeout.ms specified in the config 
to take effect like other kafka consumer properties

> Kafka Channel timeout property is overridden by default value
> -
>
> Key: FLUME-2734
> URL: https://issues.apache.org/jira/browse/FLUME-2734
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2734.patch
>
>
> Kafka Channel timeout property which will be passed to the Kafka Consumer 
> internally, does not  work as expected, as in the code, it is overridden by 
> default value (or value specified by .timeout property which is undocumented)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2660) Add documentation for EventValidator

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646818#comment-14646818
 ] 

Johny Rufus commented on FLUME-2660:


+1, Ran the -Psite and checked the UserGuide, the error is resolved, committing 

> Add documentation for EventValidator
> 
>
> Key: FLUME-2660
> URL: https://issues.apache.org/jira/browse/FLUME-2660
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.5.1
>Reporter: Hari Shreedharan
>Assignee: Ashish Paliwal
> Fix For: v1.7.0
>
> Attachments: FLUME-2660-0.patch, FLUME-2660-1.patch
>
>
> [~paliwalashish] - Assigning this to you. Please add docs for the 
> functionality you contributed in FLUME-2613



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2751) Upgrade Derby version to 10.11.1.1

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2751:
---
Attachment: FLUME-2751.patch

> Upgrade Derby version to 10.11.1.1
> --
>
> Key: FLUME-2751
> URL: https://issues.apache.org/jira/browse/FLUME-2751
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2751.patch
>
>
> Derby has lot has bug fixes in the past releases over 3 years, we should 
> upgrade to the latest stable release to 10.11.1.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2751) Upgrade Derby version to 10.11.1.1

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2751:
--

 Summary: Upgrade Derby version to 10.11.1.1
 Key: FLUME-2751
 URL: https://issues.apache.org/jira/browse/FLUME-2751
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


Derby has lot has bug fixes in the past releases over 3 years, we should 
upgrade to the latest stable release to 10.11.1.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2749) Kerberos configuration error when using short names in multiple HDFS Sinks

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2749:
---
Attachment: FLUME-2749.patch

Modified to pre-1.6 style of checking if the current user trying to log in, is 
different than the already logged in user (Using KerberosUser class, that 
stores the configured Principal and keytab)

> Kerberos configuration error when using short names in multiple HDFS Sinks
> --
>
> Key: FLUME-2749
> URL: https://issues.apache.org/jira/browse/FLUME-2749
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.6.0
>    Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2749.patch
>
>
> When we have more thank one HDFS Sink, configured in kerberos mode, and 
> principal is configured with a short name like 'flume' (without the @REALM 
> information), we get a 
> java.lang.IllegalStateException: Cannot use multiple kerberos principals in 
> the same agent.  Must restart agent to use new principal or keytab. Previous 
> = fl...@example.com (auth:KERBEROS), New = flume
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:172)
>   at 
> org.apache.flume.auth.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:131)
>   at 
> org.apache.flume.auth.FlumeAuthenticationUtil.getAuthenticator(FlumeAuthenticationUtil.java:67)
>   at 
> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:261)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2749) Kerberos configuration error when using short names in multiple HDFS Sinks

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2749:
--

 Summary: Kerberos configuration error when using short names in 
multiple HDFS Sinks
 Key: FLUME-2749
 URL: https://issues.apache.org/jira/browse/FLUME-2749
 Project: Flume
  Issue Type: Bug
Affects Versions: v1.6.0
Reporter: Johny Rufus
Assignee: Johny Rufus


When we have more thank one HDFS Sink, configured in kerberos mode, and 
principal is configured with a short name like 'flume' (without the @REALM 
information), we get a 

java.lang.IllegalStateException: Cannot use multiple kerberos principals in the 
same agent.  Must restart agent to use new principal or keytab. Previous = 
fl...@example.com (auth:KERBEROS), New = flume
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at 
org.apache.flume.auth.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:131)
at 
org.apache.flume.auth.FlumeAuthenticationUtil.getAuthenticator(FlumeAuthenticationUtil.java:67)
at 
org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:261)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641803#comment-14641803
 ] 

Johny Rufus commented on FLUME-2672:


[~rigo], I can take your patch and submit a git diff based on your changes, but 
the TestKafkaSourceCounter.java is missing the Apache License header, without 
which we wont be able to process your patch, can you kindly look at adding the 
Apache License header

> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Assignee: Rigo MacTaggart
>Priority: Trivial
>  Labels: easyfix
> Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() 
> because it expects MonitoredCounterGroup.counterMap to contain key 
> "source.kafka.empty.count". A patch is included which adds this key to the 
> ATTRIBUTES string array, which is used to pre-populate keys with an initial 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2660) Add documentation for EventValidator

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640040#comment-14640040
 ] 

Johny Rufus commented on FLUME-2660:


+1, Committing this.

> Add documentation for EventValidator
> 
>
> Key: FLUME-2660
> URL: https://issues.apache.org/jira/browse/FLUME-2660
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.5.1
>Reporter: Hari Shreedharan
>Assignee: Ashish Paliwal
> Attachments: FLUME-2660-0.patch
>
>
> [~paliwalashish] - Assigning this to you. Please add docs for the 
> functionality you contributed in FLUME-2613



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 32963: Patch for FLUME-2660 - Add documentation for EventValidator

[Johny Rufus John]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32963/#review92876
---

Ship it!


Ship It!

- Johny Rufus John


On April 8, 2015, 6:37 a.m., Ashish Paliwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32963/
> ---
> 
> (Updated April 8, 2015, 6:37 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2660
> https://issues.apache.org/jira/browse/FLUME-2660
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added documentation for Event Validator implementation
> 
> 
> Diffs
> -
> 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 576cba1 
> 
> Diff: https://reviews.apache.org/r/32963/diff/
> 
> 
> Testing
> ---
> 
> NA doc patch
> 
> 
> Thanks,
> 
> Ashish Paliwal
> 
>

[jira] [Updated] (FLUME-2672) NPE in KafkaSourceCounter

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2672:
---
Assignee: (was: Johny Rufus)

> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Priority: Trivial
>  Labels: easyfix
> Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() 
> because it expects MonitoredCounterGroup.counterMap to contain key 
> "source.kafka.empty.count". A patch is included which adds this key to the 
> ATTRIBUTES string array, which is used to pre-populate keys with an initial 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2672) NPE in KafkaSourceCounter

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2672:
---
Assignee: Rigo MacTaggart

> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Assignee: Rigo MacTaggart
>Priority: Trivial
>  Labels: easyfix
> Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() 
> because it expects MonitoredCounterGroup.counterMap to contain key 
> "source.kafka.empty.count". A patch is included which adds this key to the 
> ATTRIBUTES string array, which is used to pre-populate keys with an initial 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLUME-2672) NPE in KafkaSourceCounter

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus reassigned FLUME-2672:
--

Assignee: Johny Rufus

> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Assignee: Johny Rufus
>Priority: Trivial
>  Labels: easyfix
> Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() 
> because it expects MonitoredCounterGroup.counterMap to contain key 
> "source.kafka.empty.count". A patch is included which adds this key to the 
> ATTRIBUTES string array, which is used to pre-populate keys with an initial 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635382#comment-14635382
 ] 

Johny Rufus commented on FLUME-2672:


[~rigo], will it be possible to attach a 'git diff', instead of diffing from 
IntelliJ

> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Priority: Trivial
>  Labels: easyfix
> Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() 
> because it expects MonitoredCounterGroup.counterMap to contain key 
> "source.kafka.empty.count". A patch is included which adds this key to the 
> ATTRIBUTES string array, which is used to pre-populate keys with an initial 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2053) TestExecSource.testMonitoredCounterGroup fails

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633955#comment-14633955
 ] 

Johny Rufus commented on FLUME-2053:


[~ashishpaliwal], am able to reproduce this in my environment on JDK8, when I 
run this specific test alone, do you want me to take a stab at this one ?
(The test passes if you run any additional "runTestShellCmdHelper" inside the 
testMonitoredCounterGroup() , but not sure if this helps in trying to fix the 
root cause of the test failiure)

> TestExecSource.testMonitoredCounterGroup fails
> --
>
> Key: FLUME-2053
> URL: https://issues.apache.org/jira/browse/FLUME-2053
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: v1.4.0, v1.5.1
> Environment: Ubuntu 13.04
> Java(TM) SE Runtime Environment (build 1.7.0_13-b20)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> Maven 3.0.5
>Reporter: Hans Uhlig
>Assignee: Ashish Paliwal
>
> Failed tests:   
> testMonitoredCounterGroup(org.apache.flume.source.TestExecSource): Expected 
> Value: 5 expected:<[5]> but was:<[0]>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2628) Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2628:
---
Assignee: Lionel Herbet

> Add an optional parameter to specify the expected input text encoding for the 
> netcat sourcef the netcat source
> --
>
> Key: FLUME-2628
> URL: https://issues.apache.org/jira/browse/FLUME-2628
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Lionel Herbet
>Assignee: Lionel Herbet
>Priority: Trivial
>  Labels: newbie, patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2628.patch, FLUME-2628.patch, FLUME-2628.patch, 
> Netcat_Encoding.patch, Netcat_Encoding_WithUnitTest.patch, 
> Netcat_WithUnitTest.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> Add the 'encoding' configuration parameter to specify the expected text 
> encoding of the netcat source in input.
> This parameter is optional and defaulted to "utf-8" (same value as the one 
> hardcoded before the patch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2628) Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2628.

   Resolution: Fixed
Fix Version/s: (was: v1.5.2)
   v1.7.0

Thanks [~lionel_h] for the patch !!

> Add an optional parameter to specify the expected input text encoding for the 
> netcat sourcef the netcat source
> --
>
> Key: FLUME-2628
> URL: https://issues.apache.org/jira/browse/FLUME-2628
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Lionel Herbet
>Priority: Trivial
>  Labels: newbie, patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2628.patch, FLUME-2628.patch, FLUME-2628.patch, 
> Netcat_Encoding.patch, Netcat_Encoding_WithUnitTest.patch, 
> Netcat_WithUnitTest.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> Add the 'encoding' configuration parameter to specify the expected text 
> encoding of the netcat source in input.
> This parameter is optional and defaulted to "utf-8" (same value as the one 
> hardcoded before the patch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2628) Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631587#comment-14631587
 ] 

Johny Rufus commented on FLUME-2628:


+1, Committing the latest patch.

> Add an optional parameter to specify the expected input text encoding for the 
> netcat sourcef the netcat source
> --
>
> Key: FLUME-2628
> URL: https://issues.apache.org/jira/browse/FLUME-2628
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Lionel Herbet
>Priority: Trivial
>  Labels: newbie, patch
> Fix For: v1.5.2
>
> Attachments: FLUME-2628.patch, FLUME-2628.patch, FLUME-2628.patch, 
> Netcat_Encoding.patch, Netcat_Encoding_WithUnitTest.patch, 
> Netcat_WithUnitTest.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> Add the 'encoding' configuration parameter to specify the expected text 
> encoding of the netcat source in input.
> This parameter is optional and defaulted to "utf-8" (same value as the one 
> hardcoded before the patch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2628) Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14622753#comment-14622753
 ] 

Johny Rufus commented on FLUME-2628:


Thanks [~lionel_h], the patch works, ran tests and everything looks good, 
one more issue to be resolved before I can commit, can you include the Apache 
License Header to the new file - "TestNetcatSource.java" (sorry I missed this 
in my review)


> Add an optional parameter to specify the expected input text encoding for the 
> netcat sourcef the netcat source
> --
>
> Key: FLUME-2628
> URL: https://issues.apache.org/jira/browse/FLUME-2628
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Lionel Herbet
>Priority: Trivial
>  Labels: newbie, patch
> Fix For: v1.5.2
>
> Attachments: FLUME-2628.patch, FLUME-2628.patch, 
> Netcat_Encoding.patch, Netcat_Encoding_WithUnitTest.patch, 
> Netcat_WithUnitTest.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> Add the 'encoding' configuration parameter to specify the expected text 
> encoding of the netcat source in input.
> This parameter is optional and defaulted to "utf-8" (same value as the one 
> hardcoded before the patch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2628) Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621662#comment-14621662
 ] 

Johny Rufus commented on FLUME-2628:


[~lionel_h], the patch does not apply cleanly,
can you do a "git diff"


> Add an optional parameter to specify the expected input text encoding for the 
> netcat sourcef the netcat source
> --
>
> Key: FLUME-2628
> URL: https://issues.apache.org/jira/browse/FLUME-2628
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Lionel Herbet
>Priority: Trivial
>  Labels: newbie, patch
> Fix For: v1.5.2
>
> Attachments: FLUME-2628.patch, Netcat_Encoding.patch, 
> Netcat_Encoding_WithUnitTest.patch, Netcat_WithUnitTest.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> Add the 'encoding' configuration parameter to specify the expected text 
> encoding of the netcat source in input.
> This parameter is optional and defaulted to "utf-8" (same value as the one 
> hardcoded before the patch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2738) Async HBase sink FD leak on client shutdown

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2738:
---
Fix Version/s: v1.7.0

> Async HBase sink FD leak on client shutdown
> ---
>
> Key: FLUME-2738
> URL: https://issues.apache.org/jira/browse/FLUME-2738
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Fix For: v1.7.0
>
> Attachments: FLUME-2738-1.patch, FLUME-2738-2.patch, 
> FLUME-2738-3.patch, FLUME-2738.patch
>
>
> Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD 
> leak due to HBaseSink using CustomChannelFactory where 
> releaseExternalResources() is overridden to a No-op. Need to replace this 
> with a standard NioClientSocketChannelFactory, that releases external 
> resources properly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2738) Async HBase sink FD leak on client shutdown

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2738:
---
Attachment: FLUME-2738-3.patch

Attached, latest patch with minor changes

> Async HBase sink FD leak on client shutdown
> ---
>
> Key: FLUME-2738
> URL: https://issues.apache.org/jira/browse/FLUME-2738
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2738-1.patch, FLUME-2738-2.patch, 
> FLUME-2738-3.patch, FLUME-2738.patch
>
>
> Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD 
> leak due to HBaseSink using CustomChannelFactory where 
> releaseExternalResources() is overridden to a No-op. Need to replace this 
> with a standard NioClientSocketChannelFactory, that releases external 
> resources properly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2738) Async HBase sink FD leak on client shutdown

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2738:
---
Attachment: FLUME-2738-2.patch

> Async HBase sink FD leak on client shutdown
> ---
>
> Key: FLUME-2738
> URL: https://issues.apache.org/jira/browse/FLUME-2738
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2738-1.patch, FLUME-2738-2.patch, FLUME-2738.patch
>
>
> Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD 
> leak due to HBaseSink using CustomChannelFactory where 
> releaseExternalResources() is overridden to a No-op. Need to replace this 
> with a standard NioClientSocketChannelFactory, that releases external 
> resources properly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2738) Async HBase sink FD leak on client shutdown

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2738:
---
Attachment: FLUME-2738-1.patch

Attaching a patch that tests the FD leak. Consistently been able to reproduce 
the FD leak w/o the patch and with the patch the leaks do not happen. 
releaseExternalResources shuts down the executors which seems to impact the 
release of the sockets. 
http://netty.io/3.6/api/org/jboss/netty/channel/socket/nio/NioClientSocketChannelFactory.html
  mentions that
"All the related resources such as Selector are also released when the boss and 
worker threads are released. "


> Async HBase sink FD leak on client shutdown
> ---
>
> Key: FLUME-2738
> URL: https://issues.apache.org/jira/browse/FLUME-2738
> Project: Flume
>  Issue Type: Bug
>Reporter: Johny Rufus
>Assignee: Johny Rufus
> Attachments: FLUME-2738-1.patch, FLUME-2738.patch
>
>
> Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD 
> leak due to HBaseSink using CustomChannelFactory where 
> releaseExternalResources() is overridden to a No-op. Need to replace this 
> with a standard NioClientSocketChannelFactory, that releases external 
> resources properly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2738) Async HBase sink FD leak on client shutdown

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2738:
---
Attachment: FLUME-2738.patch

> Async HBase sink FD leak on client shutdown
> ---
>
> Key: FLUME-2738
> URL: https://issues.apache.org/jira/browse/FLUME-2738
> Project: Flume
>  Issue Type: Bug
>        Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2738.patch
>
>
> Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD 
> leak due to HBaseSink using CustomChannelFactory where 
> releaseExternalResources() is overridden to a No-op. Need to replace this 
> with a standard NioClientSocketChannelFactory, that releases external 
> resources properly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2732) Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus updated FLUME-2732:
---
Attachment: FLUME-2732.patch

Attached patch, which makes the maximum consecutive hbase failures configurable

> Make maximum tolerated failures before shutting down and recreating client in 
> AsyncHbaseSink configurable
> -
>
> Key: FLUME-2732
> URL: https://issues.apache.org/jira/browse/FLUME-2732
> Project: Flume
>  Issue Type: Bug
>    Reporter: Johny Rufus
>    Assignee: Johny Rufus
> Attachments: FLUME-2732.patch
>
>
> In AsyncHbaseSink, the maximum consecutive transaction failures, after which 
> we shutdown the HbaseClient  and recreate the client, is currently hardcoded 
> to 10. (This change was introduced to overcome a Memory leak in 
> AsyncHbaseClient)
> This needs to be configurable, and defaulted to 0 (Unlimited)
> The reason for this change is to overcome a bug in the AsyncHbaseClient, that 
> starts leaking File Descriptors on shutdown. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2738) Async HBase sink FD leak on client shutdown

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2738:
--

 Summary: Async HBase sink FD leak on client shutdown
 Key: FLUME-2738
 URL: https://issues.apache.org/jira/browse/FLUME-2738
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


Currently every time Async Hbase Sink calls HBaseSink.shutdown, there is FD 
leak due to HBaseSink using CustomChannelFactory where 
releaseExternalResources() is overridden to a No-op. Need to replace this with 
a standard NioClientSocketChannelFactory, that releases external resources 
properly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2713) Document Fault Tolerant Config parameters in FlumeUserGuide

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617841#comment-14617841
 ] 

Johny Rufus commented on FLUME-2713:


[~paliwalashish],  Thanks for committing the patch, 
Just a small observation - As per the Committers guide, commit message should 
be like 
"
Flume-XXX. Brief description of the problem.
 
(Contributor's Name via Committer's Name)

"

> Document Fault Tolerant Config parameters in FlumeUserGuide
> ---
>
> Key: FLUME-2713
> URL: https://issues.apache.org/jira/browse/FLUME-2713
> Project: Flume
>  Issue Type: Documentation
>    Reporter: Johny Rufus
>Assignee: Johny Rufus
> Fix For: v1.7.0
>
> Attachments: FLUME-2713.patch
>
>
> The following FaultTolerance related parameters in MorphlineSolrSink need to 
> be documented in Flume user guide
> FaultTolerance.IS_PRODUCTION_MODE
> FaultTolerance.IS_IGNORING_RECOVERABLE_EXCEPTIONS
> FaultTolerance.RECOVERABLE_EXCEPTION_CLASSES



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2628) Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616393#comment-14616393
 ] 

Johny Rufus commented on FLUME-2628:


[~lionel_h], can you rename the final patch as per the patch naming guidelines, 
thanks !
https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute#HowtoContribute-ProvidingPatches
"FLUME-2628.patch"

> Add an optional parameter to specify the expected input text encoding for the 
> netcat sourcef the netcat source
> --
>
> Key: FLUME-2628
> URL: https://issues.apache.org/jira/browse/FLUME-2628
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.1
>Reporter: Lionel Herbet
>Priority: Trivial
>  Labels: newbie, patch
> Fix For: v1.5.2
>
> Attachments: Netcat_Encoding.patch, 
> Netcat_Encoding_WithUnitTest.patch, Netcat_WithUnitTest.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> Add the 'encoding' configuration parameter to specify the expected text 
> encoding of the netcat source in input.
> This parameter is optional and defaulted to "utf-8" (same value as the one 
> hardcoded before the patch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2737) Documentation for Pollable Source config parameters introduced in FLUME-2729

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2737:
--

 Summary: Documentation for Pollable Source config parameters 
introduced in FLUME-2729
 Key: FLUME-2737
 URL: https://issues.apache.org/jira/browse/FLUME-2737
 Project: Flume
  Issue Type: Documentation
Reporter: Johny Rufus
Assignee: Ted Malaska


Documentation needs to be updated in Flume User Guide, for all Pollable 
Sources, that would use the new config parameters introduced in FLUME-2729



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2729) Allow pollableSource backoff times to be configurable

[Johny Rufus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johny Rufus resolved FLUME-2729.

   Resolution: Fixed
Fix Version/s: v1.7.0

Committed. Thanks Ted!

> Allow pollableSource backoff times to be configurable 
> --
>
> Key: FLUME-2729
> URL: https://issues.apache.org/jira/browse/FLUME-2729
> Project: Flume
>  Issue Type: Improvement
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2729.patch.2, FLUME-2729.patch.4, 
> FLUME-2729.patch.7, FLUME_2729.patch
>
>
> There are use cases that require more control of the backoff times setting.  
> This jira will allow those times to be set through config of the source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2729) Allow pollableSource backoff times to be configurable

[Johny Rufus (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615185#comment-14615185
 ] 

Johny Rufus commented on FLUME-2729:


+1, Will run tests and commit

> Allow pollableSource backoff times to be configurable 
> --
>
> Key: FLUME-2729
> URL: https://issues.apache.org/jira/browse/FLUME-2729
> Project: Flume
>  Issue Type: Improvement
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Attachments: FLUME-2729.patch.2, FLUME-2729.patch.4, 
> FLUME-2729.patch.7, FLUME_2729.patch
>
>
> There are use cases that require more control of the backoff times setting.  
> This jira will allow those times to be set through config of the source



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2735) Kafka Channel needs zookeeper.jar to function without errors

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2735:
--

 Summary: Kafka Channel needs zookeeper.jar to function without 
errors
 Key: FLUME-2735
 URL: https://issues.apache.org/jira/browse/FLUME-2735
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


Currently using kafka channel, gives the following error. Adding the 
zookeeper.jar (tried both 3.3.6 or 3.4.6) to the classpath helps to get past 
this error. 
(With only zkclient:jar pulled in by Kafka, the above error occurs )

2015-07-04 08:18:09,201 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - 
kafka.utils.Logging$class.info(Logging.scala:68)] 
[flume_jrufus-MBP.local-1436023089200-e4aacadf], Connecting to zookeeper 
instance at jrufus-ptest-1.vpc.cloudera.com:2181
2015-07-04 08:18:09,203 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR 
- org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to 
deliver event. Exception follows.
java.lang.IllegalStateException: close() called when transaction is OPEN - you 
must either commit or rollback first
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at 
org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
at org.apache.flume.sink.LoggerSink.process(LoggerSink.java:105)
at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2734) Kafka Channel timeout property is overridden by default value

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2734:
--

 Summary: Kafka Channel timeout property is overridden by default 
value
 Key: FLUME-2734
 URL: https://issues.apache.org/jira/browse/FLUME-2734
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


Kafka Channel timeout property which will be passed to the Kafka Consumer 
internally, does not  work as expected, as in the code, it is overridden by 
default value (or value specified by .timeout property which is undocumented)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36105: FLUME-2729 : Allow pollableSource backoff times to be configurable

[Johny Rufus John]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36105/#review90374
---

Ship it!


+1, Looks good, very minor Nits


flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
 (line 25)
<https://reviews.apache.org/r/36105/#comment143412>

Nit: Can we avoid .*



flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
 (line 33)
<https://reviews.apache.org/r/36105/#comment143413>

Nit: Not needed to specify again - implements PollableSource and 
Configurable



flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java (line 26)
<https://reviews.apache.org/r/36105/#comment143414>

Nit: .* to be changed back



flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java (line 51)
<https://reviews.apache.org/r/36105/#comment143415>

Nit: Not needed to specify again - implements PollableSource and 
Configurable



flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
 (line 72)
<https://reviews.apache.org/r/36105/#comment143417>

Not needed to specify implements Configurable and implements PollableSource


- Johny Rufus John


On July 2, 2015, 4:03 p.m., Ted Malaska wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36105/
> ---
> 
> (Updated July 2, 2015, 4:03 p.m.)
> 
> 
> Review request for Flume and Johny Rufus John.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> There are use cases that require more control of the backoff times setting. 
> This jira will allow those times to be set through config of the source
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestIntegration.java
>  4e2f940 
>   flume-ng-core/src/main/java/org/apache/flume/PollableSource.java e872b0c 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/AbstractPollableSource.java
>  356f4d4 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceConstants.java
>  PRE-CREATION 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java 
> f6c64b3 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
>  51e021a 
>   flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java 
> 0e7020b 
>   
> flume-ng-core/src/test/java/org/apache/flume/source/TestAbstractPollableSource.java
>  02a2f0c 
>   
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
>  4d4222d 
>   
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
>  c9d3e20 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestStressSource.java 
> 28270f4 
>   
> flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
>  3777639 
> 
> Diff: https://reviews.apache.org/r/36105/diff/
> 
> 
> Testing
> ---
> 
> All existing test pass.  I would like to add a new test that will confirm 
> that values are read correctly from the config.
> 
> 
> Thanks,
> 
> Ted Malaska
> 
>

[jira] [Created] (FLUME-2733) BlobDeserializer's default maxBlobLength should be less than 64 MB

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2733:
--

 Summary: BlobDeserializer's default maxBlobLength should be less 
than 64 MB
 Key: FLUME-2733
 URL: https://issues.apache.org/jira/browse/FLUME-2733
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


Currently, BlobDeserializer's default maxBlobLength is ~ 100 MB, this does not 
work very well with File Channel, as Google protobuf does not seem to handle 
messages > 64 MB. So the default should be set to ~ 60 MB 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2732) Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable

[Johny Rufus (JIRA)]

Johny Rufus created FLUME-2732:
--

 Summary: Make maximum tolerated failures before shutting down and 
recreating client in AsyncHbaseSink configurable
 Key: FLUME-2732
 URL: https://issues.apache.org/jira/browse/FLUME-2732
 Project: Flume
  Issue Type: Bug
Reporter: Johny Rufus
Assignee: Johny Rufus


In AsyncHbaseSink, the maximum consecutive transaction failures, after which we 
shutdown the HbaseClient  and recreate the client, is currently hardcoded to 
10. (This change was introduced to overcome a Memory leak in AsyncHbaseClient)
This needs to be configurable, and defaulted to 0 (Unlimited)
The reason for this change is to overcome a bug in the AsyncHbaseClient, that 
starts leaking File Descriptors on shutdown. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)