Thanks Jayesh.
It worked for me.
~Swapnil
On Fri, 1 Jun 2018, 7:10 pm Lalwani, Jayesh, <jayesh.lalw...@capitalone.com>
wrote:
> This will not work the way you have implemented it. The code that you have
> here will be called only once before the streaming query is started. Once
> the streaming query starts, this code is not called
>
>
>
> What I would do is
>
> 1. Implement a udf that calculates flourtimestamp
> 2. Add a column in df2 called “Flourtimestamp”, and populate it with
> the flour timestamp using udf.
> 3. Write the stream using
>
> Df2.writeStream.format(“text”).partitionBy(“flourtimestamp”).option(“path”,
> “/home/data”).option("checkpointLocation","./checkpoint").start()
>
>
>
> The UDF will be called for every row. And partitionBy will create a folder
> within /home/data
>
>
>
> *From: *Swapnil Chougule <the.swapni...@gmail.com>
> *Date: *Friday, June 1, 2018 at 6:21 AM
> *To: *user <user@spark.apache.org>
> *Subject: *Spark structured streaming generate output path runtime
>
>
>
> Hi
>
>
>
> I want to generate output directory runtime for data. Directory name is
> derived from current timestamp.
>
> Lets say, data for same minute should go into same directory.
>
>
>
> I tried following snippet but it didn't work. All data is being written in
> same directory (created with respect to initial timestamp)
>
>
>
> val query = df2.writeStream
> .format("text")
> .option("path", "/home/data/"+getFlourTimestamp())
> .option("checkpointLocation","./checkpoint")
> .start()
>
> query.awaitTermination()
>
>
>
> def getFlourTimestamp(): Long ={
> var curTime = System.currentTimeMillis()
> curTime - (curTime % 5000)
> }
>
>
>
>
>
> In other words, I want getFlourTimestamp() to be executed after every
> batch.
>
>
>
> Any help around this will be really appreciated.
>
>
>
> Thanks,
>
> Swapnil
>
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
