Hi,
Using: *Spark 2.3 + Kafka 0.10*
How to wait for 30 seconds after the latest stream and if there's no more
streaming data, gracefully exit.
Is it running -
query.awaitTermination(30)
Or is it something else?
I tried with this, keeping -
option("startingOffsets", "latest")
for both my input streams be joined.
Am first running the Spark job, and then pushing both the csv file data
into the two respective Kafka topics, but getting the following error -
ERROR MicroBatchExecution:91 - Query [id =
21c96e5c-770d-4d59-8893-4401217120b6, runId =
e03d7ac9-97b6-442e-adf4-dd232f9ed616] terminated with error
org.apache.spark.SparkException: Writing job aborted.
When I keep -
option("startingOffsets", "earliest")
The first batch output works perfectly and then terminates after the given
time.
Please help!
Thanks,
Aakash.
