sid created SPARK-8331:
--------------------------
Summary: DirectStream wont work
Key: SPARK-8331
URL: https://issues.apache.org/jira/browse/SPARK-8331
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.4.0
Environment: all, Kafka 0.8.2.1
Reporter: sid
I created a topic and pumped in messages using
kafka-console-producer < input file
I use receiver based stream; and it works perfect
JavaPairReceiverInputDStream<String, String> dstream =
KafkaUtils.createStream(ssc, props.getProperty("zookeeper.connect"),
props.getProperty("group.id"), topics);
But when i use Direct Stream; it just sits there hanging
JavaPairInputDStream<String, String> dstream =
KafkaUtils.createDirectStream(ssc, String.class, String.class,
StringDecoder.class, StringDecoder.class, params, topics);
my params like group.id are simply ignored
Property auto.offset.reset is overridden to smallest
Property client.id is overridden to sparkTestLocal
Property group.id is overridden to
Property zookeeper.connect is overridden to
Looking into code both KafkaCluster() using String encode/decode.
Also since we are NOT passing number of partitions; i am guess all the 10
partitions i have should be receiving in 10 threads i created
I even manually gen kafka-console-producer; while the DirectStream is running.
Just lies dead!!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
