Hi Shubham, I've just checked the latest master branch and I can confirm it works as you've described. As a workaround one can read the *<some-random-uuid>* in the directory structure and can be set with .queryName("<some-random-uuid>") before restart.
BR, G On Tue, Dec 11, 2018 at 6:45 AM Shubham Chaurasia <shubh.chaura...@gmail.com> wrote: > Hi, > > I would like to confirm checkpointing behavior, I have observed following > scenarios: > > *1)* When I set checkpointLocation from streaming query like: > > val query = > rateDF.writeStream.format("console").outputMode("append").trigger(Trigger.ProcessingTime("1 > seconds")).*option("checkpointLocation", > "/Users/shubham/checkpoint_from_query1")*.queryName("q2").start > > It generates all the metadata in */Users/shubham/checkpoint_from_query1 > *regardless > of whether queryName is set or not. > > *2)* When I set it from conf like: > *spark.conf.set("spark.sql.streaming.checkpointLocation", > "/Users/shubham/checkpoint_from_conf")* > > I observed two cases here: > *2.1)* When I set the queryName like .queryName("q2"), it generates all > metadata under */Users/shubham/checkpoint_from_conf/q2* > > *2.2)* When queryName is not set, it generates all metadata under > */Users/shubham/checkpoint_from_conf/<some-random-uuid>* > > I have seen query successfully recovers in scenario *1)* and *2.1) *which > is fine. > It does not recover from *2.2) *which is also fine as it is unable to > somehow get the query handle. > > Can there be any other possibility? Would like to confirm. > > Thanks, > Shubham > >