Re: Spark structured streaming generate output path runtime

2018-06-04 Thread Swapnil Chougule
Swapnil Chougule > *Date: *Friday, June 1, 2018 at 6:21 AM > *To: *user > *Subject: *Spark structured streaming generate output path runtime > > > > Hi > > > > I want to generate output directory runtime for data. Directory name is > derived from current timestamp. >

Re: Spark structured streaming generate output path runtime

2018-06-01 Thread Lalwani, Jayesh
d for every row. And partitionBy will create a folder within /home/data From: Swapnil Chougule Date: Friday, June 1, 2018 at 6:21 AM To: user Subject: Spark structured streaming generate output path runtime Hi I want to generate output directory runtime for data. Directory name is derived fr

Spark structured streaming generate output path runtime

2018-06-01 Thread Swapnil Chougule
Hi I want to generate output directory runtime for data. Directory name is derived from current timestamp. Lets say, data for same minute should go into same directory. I tried following snippet but it didn't work. All data is being written in same directory (created with respect to initial times