Re: What Source/Sink would you want next?
(cross-posting this to dev@) While I do not speak for the availability of other committers of the project, I would like to spend sometime with the contributors to help identify what are the most important needs of the project, and see how best we can get those committed into the codebase. Santiago (and others who would like to contribute) - please go ahead and create the necessary Jiras if they do not exist already, and invite the community to vote on those. That way we can prioritize the review and commit for functionality that is aligned with community requirements. Regards, Arvind Prabhakar On Fri, Sep 26, 2014 at 5:13 AM, jean garutti lagaru...@yahoo.fr wrote: hi This seems to be great. I'll wait to have the 'production ready' flag for ELS mapping patch. I think more effort should be done to have this sink more configurable like what we can do with logstash. anyway it's nice to share your development to the community i'd love to have the mongodb sink packaged in the official flume release. jean Le Jeudi 25 septembre 2014 9h48, Santiago Mola sm...@stratio.com a écrit : Hi Jean, 2014-09-24 22:44 GMT+02:00 Jean lagaru...@yahoo.fr: A solid mongodb source would be Nice. Definitely! I wish the same for elasticsearch sink where we could specify the mapping for the headers instead of sending everything as a string We have a serializer that creates mappings for ElasticSearch [1]. It is not ready for production [2] but it is one of our priorities. [1] https://github.com/Stratio/stratio-ingestion/tree/develop/stratio-serializers/stratio-elasticsearch-serializer [2] https://github.com/Stratio/stratio-ingestion/issues/21 Thanks for your feedback, -- Santiago M. Mola http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd https://twitter.com/StratioBD*
Re: What Source/Sink would you want next?
Strong +1. If there are new features or fixes you want to contribute - please do, I am trying to make sure I spend some time reviewing and committing. If you don’t get a review in a few days, please ping on the jira - I will look at it! Thanks, Hari On Fri, Sep 26, 2014 at 11:07 AM, Arvind Prabhakar arv...@apache.org wrote: (cross-posting this to dev@) While I do not speak for the availability of other committers of the project, I would like to spend sometime with the contributors to help identify what are the most important needs of the project, and see how best we can get those committed into the codebase. Santiago (and others who would like to contribute) - please go ahead and create the necessary Jiras if they do not exist already, and invite the community to vote on those. That way we can prioritize the review and commit for functionality that is aligned with community requirements. Regards, Arvind Prabhakar On Fri, Sep 26, 2014 at 5:13 AM, jean garutti lagaru...@yahoo.fr wrote: hi This seems to be great. I'll wait to have the 'production ready' flag for ELS mapping patch. I think more effort should be done to have this sink more configurable like what we can do with logstash. anyway it's nice to share your development to the community i'd love to have the mongodb sink packaged in the official flume release. jean Le Jeudi 25 septembre 2014 9h48, Santiago Mola sm...@stratio.com a écrit : Hi Jean, 2014-09-24 22:44 GMT+02:00 Jean lagaru...@yahoo.fr: A solid mongodb source would be Nice. Definitely! I wish the same for elasticsearch sink where we could specify the mapping for the headers instead of sending everything as a string We have a serializer that creates mappings for ElasticSearch [1]. It is not ready for production [2] but it is one of our priorities. [1] https://github.com/Stratio/stratio-ingestion/tree/develop/stratio-serializers/stratio-elasticsearch-serializer [2] https://github.com/Stratio/stratio-ingestion/issues/21 Thanks for your feedback, -- Santiago M. Mola http://www.stratio.com/ Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd https://twitter.com/StratioBD*
[jira] [Commented] (FLUME-2463) Add support for Hive and HBase datasets to DatasetSink
[ https://issues.apache.org/jira/browse/FLUME-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149787#comment-14149787 ] Hari Shreedharan commented on FLUME-2463: - +1. Committing Add support for Hive and HBase datasets to DatasetSink -- Key: FLUME-2463 URL: https://issues.apache.org/jira/browse/FLUME-2463 Project: Flume Issue Type: Improvement Reporter: Ryan Blue Assignee: Ryan Blue Attachments: FLUME-2463-1.patch, FLUME-2463-2.patch, FLUME-2463-3.patch, FLUME-2463-4.patch, FLUME-2463-5.patch, FLUME-2463-6.patch The current configuration only allows the DatasetSink to write to HDFS, but datasets commonly have hbase or hive URIs. By adding the kite-data-hive and kite-data-hbase dependencies, the DatasetSink can write to those stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2463) Add support for Hive and HBase datasets to DatasetSink
[ https://issues.apache.org/jira/browse/FLUME-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149790#comment-14149790 ] Hari Shreedharan commented on FLUME-2463: - Committed! Thanks Ryan! Add support for Hive and HBase datasets to DatasetSink -- Key: FLUME-2463 URL: https://issues.apache.org/jira/browse/FLUME-2463 Project: Flume Issue Type: Improvement Reporter: Ryan Blue Assignee: Ryan Blue Attachments: FLUME-2463-1.patch, FLUME-2463-2.patch, FLUME-2463-3.patch, FLUME-2463-4.patch, FLUME-2463-5.patch, FLUME-2463-6.patch The current configuration only allows the DatasetSink to write to HDFS, but datasets commonly have hbase or hive URIs. By adding the kite-data-hive and kite-data-hbase dependencies, the DatasetSink can write to those stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2463) Add support for Hive and HBase datasets to DatasetSink
[ https://issues.apache.org/jira/browse/FLUME-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149788#comment-14149788 ] ASF subversion and git services commented on FLUME-2463: Commit 6d0243112fa1ff1cb796ebe158630ae681a2afc8 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=6d02431 ] FLUME-2463. Add Hive and HBase dataset support in DatasetSink. (Ryan Blue via Hari) Add support for Hive and HBase datasets to DatasetSink -- Key: FLUME-2463 URL: https://issues.apache.org/jira/browse/FLUME-2463 Project: Flume Issue Type: Improvement Reporter: Ryan Blue Assignee: Ryan Blue Attachments: FLUME-2463-1.patch, FLUME-2463-2.patch, FLUME-2463-3.patch, FLUME-2463-4.patch, FLUME-2463-5.patch, FLUME-2463-6.patch The current configuration only allows the DatasetSink to write to HDFS, but datasets commonly have hbase or hive URIs. By adding the kite-data-hive and kite-data-hbase dependencies, the DatasetSink can write to those stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2463) Add support for Hive and HBase datasets to DatasetSink
[ https://issues.apache.org/jira/browse/FLUME-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149789#comment-14149789 ] ASF subversion and git services commented on FLUME-2463: Commit 89e0e53ee60840fc0a50bf62085b83384b754f9f in flume's branch refs/heads/flume-1.6 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=89e0e53 ] FLUME-2463. Add Hive and HBase dataset support in DatasetSink. (Ryan Blue via Hari) Add support for Hive and HBase datasets to DatasetSink -- Key: FLUME-2463 URL: https://issues.apache.org/jira/browse/FLUME-2463 Project: Flume Issue Type: Improvement Reporter: Ryan Blue Assignee: Ryan Blue Attachments: FLUME-2463-1.patch, FLUME-2463-2.patch, FLUME-2463-3.patch, FLUME-2463-4.patch, FLUME-2463-5.patch, FLUME-2463-6.patch The current configuration only allows the DatasetSink to write to HDFS, but datasets commonly have hbase or hive URIs. By adding the kite-data-hive and kite-data-hbase dependencies, the DatasetSink can write to those stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Jenkins build is unstable: flume-trunk #663
See https://builds.apache.org/job/flume-trunk/663/changes
[jira] [Commented] (FLUME-2463) Add support for Hive and HBase datasets to DatasetSink
[ https://issues.apache.org/jira/browse/FLUME-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149904#comment-14149904 ] Hudson commented on FLUME-2463: --- UNSTABLE: Integrated in flume-trunk #663 (See [https://builds.apache.org/job/flume-trunk/663/]) FLUME-2463. Add Hive and HBase dataset support in DatasetSink. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=6d0243112fa1ff1cb796ebe158630ae681a2afc8) * flume-ng-sinks/flume-dataset-sink/pom.xml * pom.xml Add support for Hive and HBase datasets to DatasetSink -- Key: FLUME-2463 URL: https://issues.apache.org/jira/browse/FLUME-2463 Project: Flume Issue Type: Improvement Reporter: Ryan Blue Assignee: Ryan Blue Attachments: FLUME-2463-1.patch, FLUME-2463-2.patch, FLUME-2463-3.patch, FLUME-2463-4.patch, FLUME-2463-5.patch, FLUME-2463-6.patch The current configuration only allows the DatasetSink to write to HDFS, but datasets commonly have hbase or hive URIs. By adding the kite-data-hive and kite-data-hbase dependencies, the DatasetSink can write to those stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Flume-trunk-hbase-98 #23
See https://builds.apache.org/job/Flume-trunk-hbase-98/23/changes Changes: [hshreedharan] FLUME-2463. Add Hive and HBase dataset support in DatasetSink. -- [...truncated 5219 lines...] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-legacy-sources/flume-thrift-source/pom.xml to org.apache.flume.flume-ng-legacy-sources/flume-thrift-source/1.6.0-SNAPSHOT/flume-thrift-source-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-legacy-sources/flume-thrift-source/target/flume-thrift-source-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-legacy-sources/flume-thrift-source/1.6.0-SNAPSHOT/flume-thrift-source-1.6.0-SNAPSHOT.jar [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :flume-hdfs-sink Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume legacy Thrift Source #22 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 62938 bytes Compression is 0.0% Took 0.12 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-ng-kafka-sink/pom.xml to org.apache.flume.flume-ng-sinks/flume-ng-kafka-sink/1.6.0-SNAPSHOT/flume-ng-kafka-sink-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-ng-kafka-sink/target/flume-ng-kafka-sink-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-sinks/flume-ng-kafka-sink/1.6.0-SNAPSHOT/flume-ng-kafka-sink-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume Kafka Sink #22 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 17267 bytes Compression is 0.0% Took 77 ms [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-spillable-memory-channel/pom.xml to org.apache.flume.flume-ng-channels/flume-spillable-memory-channel/1.6.0-SNAPSHOT/flume-spillable-memory-channel-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-channels/flume-spillable-memory-channel/target/flume-spillable-memory-channel-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-channels/flume-spillable-memory-channel/1.6.0-SNAPSHOT/flume-spillable-memory-channel-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume NG Spillable Memory channel #22 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 25763 bytes Compression is 0.0% Took 0.2 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-dist/pom.xml to org.apache.flume/flume-ng-dist/1.6.0-SNAPSHOT/flume-ng-dist-1.6.0-SNAPSHOT.pom Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume NG distribution #22 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 6444 bytes Compression is 0.0% Took 2.5 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-dataset-sink/pom.xml to org.apache.flume.flume-ng-sinks/flume-dataset-sink/1.6.0-SNAPSHOT/flume-dataset-sink-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sinks/flume-dataset-sink/target/flume-dataset-sink-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-sinks/flume-dataset-sink/1.6.0-SNAPSHOT/flume-dataset-sink-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume NG Kite Dataset Sink #22 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 29601 bytes Compression is 0.0% Took 0.14 sec [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-legacy-sources/pom.xml to org.apache.flume/flume-ng-legacy-sources/1.6.0-SNAPSHOT/flume-ng-legacy-sources-1.6.0-SNAPSHOT.pom Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume legacy Sources #22 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 1752 bytes Compression is 0.0% Took 77 ms [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sources/flume-twitter-source/pom.xml to org.apache.flume.flume-ng-sources/flume-twitter-source/1.6.0-SNAPSHOT/flume-twitter-source-1.6.0-SNAPSHOT.pom [JENKINS] Archiving https://builds.apache.org/job/Flume-trunk-hbase-98/ws/flume-ng-sources/flume-twitter-source/target/flume-twitter-source-1.6.0-SNAPSHOT.jar to org.apache.flume.flume-ng-sources/flume-twitter-source/1.6.0-SNAPSHOT/flume-twitter-source-1.6.0-SNAPSHOT.jar Sending artifact delta relative to Flume-trunk-hbase-98 ? Flume Twitter Source #22 Archived 2 artifacts Archive block size is 32768
[jira] [Commented] (FLUME-2463) Add support for Hive and HBase datasets to DatasetSink
[ https://issues.apache.org/jira/browse/FLUME-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149916#comment-14149916 ] Hudson commented on FLUME-2463: --- FAILURE: Integrated in Flume-trunk-hbase-98 #23 (See [https://builds.apache.org/job/Flume-trunk-hbase-98/23/]) FLUME-2463. Add Hive and HBase dataset support in DatasetSink. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=6d0243112fa1ff1cb796ebe158630ae681a2afc8) * flume-ng-sinks/flume-dataset-sink/pom.xml * pom.xml Add support for Hive and HBase datasets to DatasetSink -- Key: FLUME-2463 URL: https://issues.apache.org/jira/browse/FLUME-2463 Project: Flume Issue Type: Improvement Reporter: Ryan Blue Assignee: Ryan Blue Attachments: FLUME-2463-1.patch, FLUME-2463-2.patch, FLUME-2463-3.patch, FLUME-2463-4.patch, FLUME-2463-5.patch, FLUME-2463-6.patch The current configuration only allows the DatasetSink to write to HDFS, but datasets commonly have hbase or hive URIs. By adding the kite-data-hive and kite-data-hbase dependencies, the DatasetSink can write to those stores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2479) Kafka property auto.commit.enable is incorrect for KafkaSource
Hari Shreedharan created FLUME-2479: --- Summary: Kafka property auto.commit.enable is incorrect for KafkaSource Key: FLUME-2479 URL: https://issues.apache.org/jira/browse/FLUME-2479 Project: Flume Issue Type: Bug Reporter: Hari Shreedharan The KafkaSource uses auto.commit.enabled (the d should be removed) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
