[ https://issues.apache.org/jira/browse/CONNECTORS-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692449#comment-14692449 ]
Tugba Dogan commented on CONNECTORS-1162: ----------------------------------------- Hi Karl, I've been struggling with Kafka consumer for 2 weeks. I was looking this documentation to implement Kafka Consumer: http://kafka.apache.org/083/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html However, `consumer.poll();` method were returning null always and not throwing any exception. Then I realized that they haven't implement the functions yet, they're planning to release implemented version in October 2015 :) Today I found Kafka's scala based library which requires extra 2-3 dependency to work properly. Finally, I'm able to consume messages from Kafka from manifold. In 1-2 days I'll complete recording Kafka messages to Manifold. > Apache Kafka Output Connector > ----------------------------- > > Key: CONNECTORS-1162 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1162 > Project: ManifoldCF > Issue Type: Wish > Affects Versions: ManifoldCF 1.8.1, ManifoldCF 2.0.1 > Reporter: Rafa Haro > Assignee: Karl Wright > Labels: gsoc, gsoc2015 > Fix For: ManifoldCF 2.3 > > Attachments: 1.JPG, 2.JPG > > > Kafka is a distributed, partitioned, replicated commit log service. It > provides the functionality of a messaging system, but with a unique design. A > single Kafka broker can handle hundreds of megabytes of reads and writes per > second from thousands of clients. > Apache Kafka is being used for a number of uses cases. One of them is to use > Kafka as a feeding system for streaming BigData processes, both in Apache > Spark or Hadoop environment. A Kafka output connector could be used for > streaming or dispatching crawled documents or metadata and put them in a > BigData processing pipeline -- This message was sent by Atlassian JIRA (v6.3.4#6332)