[jira] [Created] (FLUME-2204) Expose Peer SSL Certificate in AvroSource

2013-10-03 Thread Charles Pritchard (JIRA) 
Charles Pritchard created FLUME-2204:


 Summary: Expose Peer SSL Certificate in AvroSource
 Key: FLUME-2204
 URL: https://issues.apache.org/jira/browse/FLUME-2204
 Project: Flume
  Issue Type: Bug
Reporter: Charles Pritchard


AvroSource supports SSL connections, getPeerCertificateChain()[0] will expose 
the client SSL certificate. Through that getSubjectDN().getName() gives us very 
usable information.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (FLUME-2191) HDFS Minicluster tests failing after protobuf upgrade.

2013-10-03 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785863#comment-13785863
 ] 

Hudson commented on FLUME-2191:
---

UNSTABLE: Integrated in flume-trunk #508 (See 
[https://builds.apache.org/job/flume-trunk/508/])
FLUME-2191. HDFS Minicluster tests failing after protobuf upgrade. (arvind: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9c59a309764498c013ccd202926d86413da01078)
* flume-ng-sinks/flume-hdfs-sink/pom.xml
* pom.xml


> HDFS Minicluster tests failing after protobuf upgrade.
> --
>
> Key: FLUME-2191
> URL: https://issues.apache.org/jira/browse/FLUME-2191
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
>Priority: Blocker
> Fix For: v1.5.0
>
> Attachments: FLUME-2191.patch
>
>
> I ran the full build in hadoop-1 profile, but it looks like the protobuf 
> upgrade broke the hadoop-2 profile. The HDFS Sink test on Minicluster fails 
> with this:
> {code}
> Running org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster
> 2013-09-13 12:11:31.159 java[58566:1203] Unable to load realm info from 
> SCDynamicStore
> 2013-09-13 12:11:31.208 java[58566:1203] Unable to load realm info from 
> SCDynamicStore
> Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 4.238 sec <<< 
> FAILURE!
> simpleHDFSTest(org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  
> Time elapsed: 1979 sec  <<< ERROR!
> java.lang.UnsupportedOperationException: This is supposed to be overridden by 
> subclasses.
>   at 
> com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetDatanodeReportRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:21638)
>   at 
> com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:137)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:181)
>   at com.sun.proxy.$Proxy15.getDatanodeReport(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
>   at com.sun.proxy.$Proxy15.getDatanodeReport(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:488)
>   at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1642)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1703)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1722)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1066)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:929)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:588)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:527)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:398)
>   at 
> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.simpleHDFSTest(TestHDFSEventSinkOnMiniCluster.java:85)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner

Jenkins build is unstable: flume-trunk #508

2013-10-03 Thread Apache Jenkins Server 
See

[jira] [Commented] (FLUME-2202) AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785800#comment-13785800
 ] 

Hudson commented on FLUME-2202:
---

FAILURE: Integrated in flume-trunk #507 (See 
[https://builds.apache.org/job/flume-trunk/507/])
FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC roundtrips 
(mpercy: 
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c4e2129fd12f97303a1b8120a2ecf7da456e1b77)
* 
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java
* 
flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
* 
flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
* flume-ng-doc/sphinx/FlumeUserGuide.rst
* 
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java


> AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
> --
>
> Key: FLUME-2202
> URL: https://issues.apache.org/jira/browse/FLUME-2202
> Project: Flume
>  Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2202-2.patch, FLUME-2202-3.patch, FLUME-2202.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Build failed in Jenkins: flume-trunk #507

2013-10-03 Thread Apache Jenkins Server 
See 

Changes:

[mpercy] FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC 
roundtrips

--
[...truncated 761 lines...]
[INFO] Exclude: **/*.patch
[INFO] Exclude: **/*.avsc
[INFO] Exclude: **/*.avro
[INFO] Exclude: **/docs/**
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: **/.settings/*
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.project
[INFO] Exclude: **/target/**
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: 
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: 
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: Building Flume NG Channels 1.5.0-SNAPSHOT
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: 
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-clean-plugin:2.4.1:clean (default-clean) @ flume-ng-channels ---
[INFO] Deleting 

Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-remote-resources-plugin:1.1:process (default) @ 
flume-ng-channels ---
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:15:58 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- apache-rat-plugin:0.7:check (verify.rat) @ flume-ng-channels ---
[INFO] Exclude: .idea/
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/nb-configuration.xml
[INFO] Exclude: .git/
[INFO] Exclude: patchprocess/
[INFO] Exclude: .gitignore
[INFO] Exclude: .repository/
[INFO] Exclude: **/*.diff
[INFO] Exclude: **/*.patch
[INFO] Exclude: **/*.avsc
[INFO] Exclude: **/*.avro
[INFO] Exclude: **/docs/**
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: **/.settings/*
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.project
[INFO] Exclude: **/target/**
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: 
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: 
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: Building Flume NG JDBC channel 1.5.0-SNAPSHOT
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
projectStarted
INFO: 
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-clean-plugin:2.4.1:clean (default-clean) @ flume-jdbc-channel 
---
[INFO] Deleting 

Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:16:03 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-remote-resources-plugin:1.1:process (default) @ 
flume-jdbc-channel ---
Oct 4, 2013 1:16:04 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:16:04 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-resources-plugin:2.4.3:resources (default-resources) @ 
flume-jdbc-channel ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
Oct 4, 2013 1:16:04 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:16:04 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-compiler-plugin:2.3.2:compile (default-compile) @ 
flume-jdbc-channel ---
[INFO] Compiling 15 source files to 

Oct 4, 2013 1:16:05 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: 
Oct 4, 2013 1:16:05 AM org.apache.maven.cli.event.ExecutionEventLogger 
mojoStarted
INFO: --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ 
flume-jdbc-channel ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing res

[jira] [Commented] (FLUME-2191) HDFS Minicluster tests failing after protobuf upgrade.

2013-10-03 Thread Arvind Prabhakar (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785791#comment-13785791
 ] 

Arvind Prabhakar commented on FLUME-2191:
-

+1 changes look good to me. Will commit after a sanity run.

> HDFS Minicluster tests failing after protobuf upgrade.
> --
>
> Key: FLUME-2191
> URL: https://issues.apache.org/jira/browse/FLUME-2191
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
>Priority: Blocker
> Attachments: FLUME-2191.patch
>
>
> I ran the full build in hadoop-1 profile, but it looks like the protobuf 
> upgrade broke the hadoop-2 profile. The HDFS Sink test on Minicluster fails 
> with this:
> {code}
> Running org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster
> 2013-09-13 12:11:31.159 java[58566:1203] Unable to load realm info from 
> SCDynamicStore
> 2013-09-13 12:11:31.208 java[58566:1203] Unable to load realm info from 
> SCDynamicStore
> Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 4.238 sec <<< 
> FAILURE!
> simpleHDFSTest(org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  
> Time elapsed: 1979 sec  <<< ERROR!
> java.lang.UnsupportedOperationException: This is supposed to be overridden by 
> subclasses.
>   at 
> com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetDatanodeReportRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:21638)
>   at 
> com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:137)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:181)
>   at com.sun.proxy.$Proxy15.getDatanodeReport(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
>   at com.sun.proxy.$Proxy15.getDatanodeReport(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:488)
>   at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1642)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1703)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1722)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1066)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:929)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:588)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:527)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:398)
>   at 
> org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.simpleHDFSTest(TestHDFSEventSinkOnMiniCluster.java:85)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentR

[jira] [Updated] (FLUME-2202) AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Mike Percy (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Percy updated FLUME-2202:
--

Release Note:   (was: Pushed to trunk and flume-1.5 branches. Thanks for 
the patch Hari!)

Pushed to trunk and flume-1.5 branches. Thanks for the patch Hari!

> AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
> --
>
> Key: FLUME-2202
> URL: https://issues.apache.org/jira/browse/FLUME-2202
> Project: Flume
>  Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.5.0
>
> Attachments: FLUME-2202-2.patch, FLUME-2202-3.patch, FLUME-2202.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (FLUME-2202) AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Mike Percy (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785722#comment-13785722
 ] 

Mike Percy commented on FLUME-2202:
---

+1

> AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
> --
>
> Key: FLUME-2202
> URL: https://issues.apache.org/jira/browse/FLUME-2202
> Project: Flume
>  Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2202-2.patch, FLUME-2202-3.patch, FLUME-2202.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 14454: FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Mike Percy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14454/#review26657
---

Ship it!


+1, looks good

- Mike Percy


On Oct. 3, 2013, 8:07 p.m., Hari Shreedharan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14454/
> ---
> 
> (Updated Oct. 3, 2013, 8:07 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2202
> https://issues.apache.org/jira/browse/FLUME-2202
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added a new config to coalesce increments. 
> 
> 
> Diffs
> -
> 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 5a59b56 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
>  5e297b1 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
>  7fdc75b 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
>  a0c04eb 
> 
> Diff: https://reviews.apache.org/r/14454/diff/
> 
> 
> Testing
> ---
> 
> All current tests pass. Added 2 new tests
> 
> 
> Thanks,
> 
> Hari Shreedharan
> 
>

A couple of features regarding ElasticSearchSink (ESS)

2013-10-03 Thread Dibyajyoti Ghosh 
Hi,

I am using flume ElasticSearch (ES) sink for my project. The flume version
being:

Flume 1.4.0-cdh4.4.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 154d35659212f07edc896b414a43996fb8121773
Compiled by jenkins on Tue Sep  3 20:53:28 PDT 2013
>From source with checksum f95b4a7f48080f876d6482bb88bcc342

And ElasticSearch v0.90.1.

I am having two issues with the current set of ES configurations allowed
from flume agent.conf

*agent.sinks.myESsink.indexName = myIndex*
*
*
and

*agent.sinks.myESsink.ttl = .*
*
*
ElasticSearchSink uses the provided index name as index prefix and appends
"-MM-DD" to generate the actual index in ES which being convenient for
my testing purposes, doesn't allow creating index monthly / yearly or more
generally speaking based on some regex provided in flume config similar to
HDFS fileSuffix .e.g.

*agent.sinks.myESsink.indexSuffix = ""* will create index as
myIndex-2013 / myIndex-2014 etc and when not provided will create index
with just the index name or can default back to '-MM-DD'.

The second one is comparatively trivial but good to have. Current ElasticSearch
TTL defaults to 5 days and works with integers only again which is treated
as days.

It will be good to have a qualifier like "d" / "s" / "m" / "w" / "h" to
mimic the TTL conf in ElasticSearch mapping.

For the second case I have already made changes in my local flume git repo
and currently testing it.

I will start working on the index naming one shortly once I get the easier
of the issues fixed and running in my local deployments.

I didn't find any JIRA tickets for these requirements in Flume Jira and was
wondering how to get these changes in the central flume code base which
will alleviate the pain of maintaining a local flume development branch
while my requirements seemingly has broad applicability.

Please suggest how should I proceed and thank you for bearing with this
long email.

Best,
- Dib

Re: Review Request 14454: FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Hari Shreedharan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14454/
---

(Updated Oct. 3, 2013, 8:07 p.m.)


Review request for Flume.


Bugs: FLUME-2202
https://issues.apache.org/jira/browse/FLUME-2202


Repository: flume-git


Description
---

Added a new config to coalesce increments. 


Diffs (updated)
-

  flume-ng-doc/sphinx/FlumeUserGuide.rst 5a59b56 
  
flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
 5e297b1 
  
flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
 7fdc75b 
  
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java
 PRE-CREATION 
  
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
 a0c04eb 

Diff: https://reviews.apache.org/r/14454/diff/


Testing
---

All current tests pass. Added 2 new tests


Thanks,

Hari Shreedharan

[jira] [Updated] (FLUME-2202) AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Hari Shreedharan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan updated FLUME-2202:


Attachment: FLUME-2202-3.patch

> AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
> --
>
> Key: FLUME-2202
> URL: https://issues.apache.org/jira/browse/FLUME-2202
> Project: Flume
>  Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Attachments: FLUME-2202-2.patch, FLUME-2202-3.patch, FLUME-2202.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 14454: FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

2013-10-03 Thread Hari Shreedharan 


> On Oct. 3, 2013, 6:34 a.m., Mike Percy wrote:
> > flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java,
> >  line 287
> > 
> >
> > I wonder if this could be broken into 2 or 3 functions in order to be 
> > easier to follow. If not easy to do that it's not a big deal.

The whole test file needs some refactoring - there is quite a bit of repeated 
code. I will file a follow up for this.


> On Oct. 3, 2013, 6:34 a.m., Mike Percy wrote:
> > flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java,
> >  line 298
> > 
> >
> > Nit: Would be better to use 
> > org.apache.flume.sink.hbase.IncrementAsyncHBaseSerializer.class.getName() 
> > so that this is refactorable and shadable.

Done.


> On Oct. 3, 2013, 6:34 a.m., Mike Percy wrote:
> > flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java,
> >  line 310
> > 
> >
> > I'm confused why we are using the same Context object to configure both 
> > a Sink and a Channel. Probably a minor bug.

Seems like that is done through the entire test. I will fix it in the follow up 
jira mentioned above.


- Hari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14454/#review26641
---


On Oct. 3, 2013, 5:32 a.m., Hari Shreedharan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14454/
> ---
> 
> (Updated Oct. 3, 2013, 5:32 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2202
> https://issues.apache.org/jira/browse/FLUME-2202
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added a new config to coalesce increments. 
> 
> 
> Diffs
> -
> 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 5a59b56 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
>  5e297b1 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
>  7fdc75b 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java
>  PRE-CREATION 
>   
> flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
>  a0c04eb 
> 
> Diff: https://reviews.apache.org/r/14454/diff/
> 
> 
> Testing
> ---
> 
> All current tests pass. Added 2 new tests
> 
> 
> Thanks,
> 
> Hari Shreedharan
> 
>

Re: Review Request 14439: Syslog source strips timestamp and hostname from log message body

2013-10-03 Thread Mike Percy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14439/#review26648
---



flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java


Style: extra line here.



flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java


The default (false) should also be specified in 
SyslogSourceConfigurationConstants



flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java


Please mark this class as @InterfaceAudience.Private and 
@InterfaceStability.Evolving



flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java


This should be SyslogSourceConfigurationConstants.DEFAULT_KEEP_FIELDS



flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java


Style: Use camel-caps: keepTimeAndHost would be the right casing. However, 
you should just keep the names consistent and make it keepFields.




flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java


Style: Would be better to say this.keepFields = keepFields



flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java


Style: The brace should stay on the same line as the conditional. If the 
line was too long to fit the brace (>80 chars), then keep the brace at the same 
indentation level as the "if".



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


Don't hard-code the port... multiple tests running at the same time (maybe 
multiple builds on the same Jenkins box) will fail. Start up the service on 
port 0 and get the kernel-selected port from the running service for use in the 
test.



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


Style: Remove this extraneous line unless you want to document the return 
param (I think it's obvious).



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


Style: Come up with a better name than doTest. How about 
runKeepFieldsTest() ?

Also, please add a comment here documenting what this test does. Something 
along the lines of: // Tests the keepFields configuration parameter (enabled or 
disabled) using SyslogTcpSource.



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


Style: This comment should be indented.



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


Don't catch this, just let testKeepFields throw IOException. It's a test 
and if it throws it will fail, which is fine.



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


e.printStackTrace() is considered bad practice these days because it can 
easily go un-noticed. You should almost always LOG an exception like this or 
rethrow it. But in this case you can just remove the whole try/catch block and 
not worry about it.



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java


Same thing here, remove the try/catch.



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java


Style: Line continuations like this are usually indented 4 spaces instead 
of 2.



flume-ng-doc/sphinx/FlumeUserGuide.rst


Style: match the indentation of this line to the word "Setting" above.


Looks good overall, just a few stylistic suggestions for the most part.

- Mike Percy


On Oct. 2, 2013, 1:04 a.m., Jeff jlord wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14439/
> ---
> 
> (Updated Oct. 2, 2013, 1:04 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Attaching a patch which introduces a boolean keepFields which defaults to 
> false. When set to true this will preserve the timestamp and hostname in the 
> body of the event. Additionally I have added a test for SyslogTcpSource
> 
> 
> Diffs
> -
> 
>   
> flume-ng-core/src/main/java/org/apache