Hi,
I would like to confirm checkpointing behavior, I have observed following
scenarios:
*1)* When I set checkpointLocation from streaming query like:
val query =
rateDF.writeStream.format("console").outputMode("append").trigger(Trigger.ProcessingTime("1
seconds")).*option("checkpointLocation",
"/Users/shubham/checkpoint_from_query1")*.queryName("q2").start
It generates all the metadata in
*/Users/shubham/checkpoint_from_query1 *regardless
of whether queryName is set or not.
*2)* When I set it from conf like:
*spark.conf.set("spark.sql.streaming.checkpointLocation",
"/Users/shubham/checkpoint_from_conf")*
I observed two cases here:
*2.1)* When I set the queryName like .queryName("q2"), it generates all
metadata under */Users/shubham/checkpoint_from_conf/q2*
*2.2)* When queryName is not set, it generates all metadata under
*/Users/shubham/checkpoint_from_conf/<some-random-uuid>*
I have seen query successfully recovers in scenario *1)* and *2.1) *which
is fine.
It does not recover from *2.2) *which is also fine as it is unable to
somehow get the query handle.
Can there be any other possibility? Would like to confirm.
Thanks,
Shubham
