[ https://issues.apache.org/jira/browse/FLUME-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187668#comment-14187668 ]
Hudson commented on FLUME-2500: ------------------------------- UNSTABLE: Integrated in flume-trunk #683 (See [https://builds.apache.org/job/flume-trunk/683/]) FLUME-2500: Add a channel that uses Kafka (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=aef02df10a26a6b6911b771a506994f2069857cf) * flume-ng-channels/flume-kafka-channel/src/test/resources/zookeeper.properties * flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java * flume-ng-channels/flume-kafka-channel/src/test/resources/kafka-server.properties * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/util/KafkaConsumer.java * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/util/TestUtil.java * flume-ng-channels/pom.xml * flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java * flume-ng-sinks/flume-ng-kafka-sink/pom.xml * flume-ng-channels/flume-kafka-channel/pom.xml * flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java * flume-ng-channels/flume-kafka-channel/src/test/resources/log4j.properties > Add a channel that uses Kafka > ------------------------------ > > Key: FLUME-2500 > URL: https://issues.apache.org/jira/browse/FLUME-2500 > Project: Flume > Issue Type: Bug > Reporter: Hari Shreedharan > Assignee: Hari Shreedharan > Fix For: v1.6.0 > > Attachments: FLUME-2500-1.patch, FLUME-2500-2.patch, > FLUME-2500.patch, FLUME-2500.patch > > > Here is the rationale: > - Kafka does give a HA channel, which means a dead agent does not affect the > data in the channel - thus reducing delay of delivery. > - Kafka is used by many companies - it would be a good idea to use Flume to > pull data from Kafka and write it to HDFS/HBase etc. > This channel is not going to be useful for cases where Kafka is not already > used, since it brings is operational overhead of maintaining two systems, but > if there is Kafka in use - this is good way to integrate Kafka and Flume. > Here is an a scratch implementation: > https://github.com/harishreedharan/flume/blob/kafka-channel/flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)