Re: Running into the same problem as JIRA SPARK-20325

2017-05-31 Thread Michael Armbrust 
>
> So, my question is the same as stated in the following ticket which is Do
> we need create a checkpoint directory for each individual query?
>

Yes.  Checkpoints record what data has been processed.  Thus two different
queries need their own checkpoints.

Running into the same problem as JIRA SPARK-20325

2017-05-31 Thread kant kodali 
Hi All,

I am using Spark 2.1.1 and forEachSink to write to Kafka. I call .start and
.awaitTermination for each query however I get the following error

"Cannot start query with id d4b3554d-ee1d-469c-bf9d-19976c8a7b47 as another
query with same id is already active"


So, my question is the same as stated in the following ticket which is Do
we need create a checkpoint directory for each individual query?

https://issues.apache.org/jira/browse/SPARK-20325

Thanks!