Dan Dutrow created SPARK-12103: ---------------------------------- Summary: KafkaUtils createStream with multiple topics -- does not work as expected Key: SPARK-12103 URL: https://issues.apache.org/jira/browse/SPARK-12103 Project: Spark Issue Type: Improvement Components: Streaming Affects Versions: 1.0.0 Reporter: Dan Dutrow Fix For: 1.0.1
Default way of creating stream out of Kafka source would be as val stream = KafkaUtils.createStream(ssc,"localhost:2181","logs", Map("retarget" -> 2,"datapair" -> 2)) However, if two topics - in this case "retarget" and "datapair" - are very different, there is no way to set up different filter, mapping functions, etc), as they are effectively merged. However, instance of KafkaInputDStream, created with this call internally calls ConsumerConnector.createMessageStream() which returns *map* of KafkaStreams, keyed by topic. It would be great if this map would be exposed somehow, so aforementioned call val streamS = KafkaUtils.createStreamS(...) returned map of streams. Regards, Sergey Malov Collective Media -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org