Spark structured streaming generate output path runtime

[Swapnil Chougule]

Hi

I want to generate output directory runtime for data. Directory name is
derived from current timestamp.
Lets say, data for same minute should go into same directory.

I tried following snippet but it didn't work. All data is being written in
same directory (created with respect to initial timestamp)

val query = df2.writeStream
      .format("text")
      .option("path", "/home/data/"+getFlourTimestamp())
      .option("checkpointLocation","./checkpoint")
      .start()

  query.awaitTermination()



  def getFlourTimestamp(): Long ={
    var curTime = System.currentTimeMillis()
    curTime - (curTime % 5000)
  }


In other words, I want getFlourTimestamp() to be executed after every batch.

Any help around this will be really appreciated.

Thanks,
Swapnil