Re: Behavior of checkpointLocation from options vs setting conf spark.sql.streaming.checkpointLocation

2018-12-12 Thread Shubham Chaurasia
Thanks Gabor. On Wed, Dec 12, 2018, 4:06 PM Gabor Somogyi Hi Shubham, > > I've just checked the latest master branch and I can confirm it works as > you've described. > As a workaround one can read the ** in the directory > structure and can be set with .queryName("") before > restart. > > BR, >

Re: Behavior of checkpointLocation from options vs setting conf spark.sql.streaming.checkpointLocation

2018-12-12 Thread Gabor Somogyi
Hi Shubham, I've just checked the latest master branch and I can confirm it works as you've described. As a workaround one can read the ** in the directory structure and can be set with .queryName("") before restart. BR, G On Tue, Dec 11, 2018 at 6:45 AM Shubham Chaurasia wrote: > Hi, > > I

Behavior of checkpointLocation from options vs setting conf spark.sql.streaming.checkpointLocation

2018-12-10 Thread Shubham Chaurasia
Hi, I would like to confirm checkpointing behavior, I have observed following scenarios: *1)* When I set checkpointLocation from streaming query like: val query = rateDF.writeStream.format("console").outputMode("append").trigger(Trigger.ProcessingTime("1 seconds")).*option("checkpointLocation",