Hello, we are glad to announce yet another kafka input stream.
Available at https://github.com/wgnet/spark-kafka-streaming It is used in production for about 3 months. We will be happy to hear your feedback. Custom Spark Kafka consumer based on Kafka SimpleConsumer API. Features • discover kafka metadata from zookeeper (more reliable than from brokers, does not depend on broker list changes) • reding from multiple topics • reliably handles leader election and topic reassignment • saves offsets and stream metadata in hbase (more robust than zookeeper) • supports metrics via spark metrics mechanism (jmx, graphite, etc.) Todo • abstract offset storage • time controlled offsets commit • refactor kafka message to rdd elements transformation (flatmapper method) --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
