t;>>>>> Since those 3 topics contain completely different data on which I
>>>>>> need to apply different kind of transformations, I am not sure joining
>>>>>> them
>>>>>> would be really efficient, unless you know something that I don&#
, unless you know something that I don't.
>>>>>
>>>>> As I really don't need any interaction between those streams, I think
>>>>> I might end up running 3 different streaming apps instead of one.
>>>>>
>>>>> Tha
;>> wrote:
>>>>
>>>>> Using spark.streaming.concurrentJobs for this probably isn't a good
>>>>> idea, as it allows the next batch to start processing before current one
>>>>> is
>>>>> finished, which may have unintende
g before current one is
>>>> finished, which may have unintended consequences.
>>>>
>>>> Why can't you use a single stream with all the topics you care about,
>>>> or multiple streams if you're e.g. joining them?
>>>>
>>>>
&g
;>
>>>>> On Wed, Dec 16, 2015 at 3:00 PM, jpocalan wrote:
>>>>> Nevermind, I found the answer to my questions.
>>>>> The following spark configuration property will allow you to process
>>>>> multiple KafkaDirectStream in parallel:
>&
d, Dec 16, 2015 at 3:00 PM, jpocalan wrote:
>>>
>>>> Nevermind, I found the answer to my questions.
>>>> The following spark configuration property will allow you to process
>>>> multiple KafkaDirectStream in par
>>> The following spark configuration property will allow you to process
>>> multiple KafkaDirectStream in parallel:
>>> --conf spark.streaming.concurrentJobs=
>>>
>>>
>>>
>>>
>>&g
parallel:
>> --conf spark.streaming.concurrentJobs=
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25723.html
>> Sent from the
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25723.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
-streaming-from-multiple-topics-tp8678p25723.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h
t;> On Wed, Dec 2, 2015 at 2:13 PM, dutrow wrote:
>>
>>> I found the JIRA ticket:
>>> https://issues.apache.org/jira/browse/SPARK-2388
>>>
>>> It was marked as invalid.
>>>
>>>
2015 at 2:13 PM, dutrow wrote:
>
>> I found the JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2388
>>
>> It was marked as invalid.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabbl
://kafka.apache.org/documentation.html#compaction
I don't see a way around this without hacking KafkaStream and
ConsumerIterator to return the topic name instead of the message key.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-t
>
>> It was marked as invalid.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25550.html
s invalid.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25550.html
> Sent from the Apache Spark User List mail
I found the JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2388
It was marked as invalid.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25550.html
Sent from the Apache Spark User List mailing list
this issue?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25549.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---
Can you link to the JIRA issue? I'm having to work around this bug and it
would be nice to monitor the JIRA so I can change my code when it's fixed.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p12053
ursday, July 3, 2014 at 9:41 PM
To: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
Subject: Re: Kafka - streaming from multiple topics
Sergey,
On Fri, Jul 4, 2014 at 1:06 AM, Sergey Malov
mailto:sma...@collective.com>> wrote:
On t
Sergey,
On Fri, Jul 4, 2014 at 1:06 AM, Sergey Malov wrote:
>
> On the other hand, under the hood KafkaInputDStream which is create with
> this KafkaUtils call, calls ConsumerConnector.createMessageStream which
> returns a Map[String, List[KafkaStream] keyed by topic. It is, however, not
> expo
...@preferred.jp>>
Reply-To: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
Date: Wednesday, July 2, 2014 at 9:54 PM
To: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
S
Sergey,
you might actually consider using two streams, like
* val stream1 = KafkaUtils.createStream(ssc,"localhost:2181","logs",
Map("retarget" -> 2))*
* val stream2 = KafkaUtils.createStream(ssc,"localhost:2181","logs",
Map("datapair" -> 2))*
to achieve what you want. This has the addit
HI,
I would like to set up streaming from Kafka cluster, reading multiple topics
and then processing each of the differently.
So, I’d create a stream
val stream = KafkaUtils.createStream(ssc,"localhost:2181","logs",
Map("retarget" -> 2,"datapair" -> 2))
And then based on whether it’s “ret
23 matches
Mail list logo
