[
https://issues.apache.org/jira/browse/SPARK-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258597#comment-14258597
]
Tathagata Das edited comment on SPARK-2388 at 12/25/14 12:26 AM:
-----------------------------------------------------------------
You can always create multiple kafka streams with different topics assigned to
them and then transform (map, filter, etc.) them differently. I am closing this
issue.
was (Author: tdas):
You can always create multiple kafka streams with different topics assigned to
them and then transform (map, filter, etc.) them differently.
> Streaming from multiple different Kafka topics is problematic
> -------------------------------------------------------------
>
> Key: SPARK-2388
> URL: https://issues.apache.org/jira/browse/SPARK-2388
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.0.0
> Reporter: Sergey
> Fix For: 1.0.1
>
>
> Default way of creating stream out of Kafka source would be as
> val stream = KafkaUtils.createStream(ssc,"localhost:2181","logs",
> Map("retarget" -> 2,"datapair" -> 2))
> However, if two topics - in this case "retarget" and "datapair" - are very
> different, there is no way to set up different filter, mapping functions,
> etc), as they are effectively merged.
> However, instance of KafkaInputDStream, created with this call internally
> calls ConsumerConnector.createMessageStream() which returns *map* of
> KafkaStreams, keyed by topic. It would be great if this map would be exposed
> somehow, so aforementioned call
> val streamS = KafkaUtils.createStreamS(...)
> returned map of streams.
> Regards,
> Sergey Malov
> Collective Media
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
