Hi All,
I have the following code.
val ds = sparkSession.readStream()
.format("kafka")
.option("kafka.bootstrap.servers",bootstrapServers))
.option("subscribe", topicName)
.option("checkpointLocation", hdfsCheckPointDir)
.load();
val ds1 = ds.select($"value")
val query = ds1.writeStream.outputMode("append").format("console").start()
query.awaitTermination()
There are no errors when I execute this code however I don't see any data
being printed out to console? When I run my standalone test Kafka consumer
jar I can see that it is receiving messages. so I am not sure what is going
on with above code? any ideas?
Thanks!