Hi All,
I am consuming a 8 partition kafka topic through multiple Dstreams and
Processing them in Spark.
But irrespective of multiple InputDstreams the spark master UI is showing
only one receiver.
The following is the consumer part of spark code:
int numStreams = 8;
List<JavaPairDStream<String, String>> kafkaStreams = new
ArrayList<JavaPairDStream<String, String>>(
numStreams);
for (int i = 0; i < numStreams; i++) {
kafkaStreams.add(KafkaUtils.createStream(jssc,
"bhucloud04.adxyz.com:2181",
"test-consumer-group",
topicMap));
}
JavaPairDStream<String, String> unifiedStream = jssc.union(
kafkaStreams.get(0),
kafkaStreams.subList(1, kafkaStreams.size()));
unifiedStream.repartition(8);
I have attached the screen shot of
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n20934/spark-kafka.png>
spark ui
Thanks
Tapas
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Only-One-Kafka-receiver-is-running-in-spark-irrespective-of-multiple-DStreams-tp20934.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]