Re: [EXTERNAL] Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Yap I was able to get this to work with a custom bucketer. A custom bucketer can use the clock given ("processing time") or it can use a timestamp from the data ("event time") for the bucketing path. From: Raja.Aravapalli Sent:

Re: [EXTERNAL] Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Thanks Aljoscha for the inputs. I will check to extend “BasePathBucketer” class. Regards, Raja. From: Aljoscha Krettek Date: Friday, September 1, 2017 at 10:27 AM To: Piotr Nowojski Cc: Raja Aravapalli ,

Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Hi Raja, I think you can in fact do this by implementing a custom Bucketer. You can have a look at BasePathBucketer and extend that to include the timestamp in the path that is returned. You should probably clamp the timestamp so that you don't get a new path for every millisecond. Best,

Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Hi, BucketingSink doesn’t support the feature that you are requesting, you can not specify a dynamically generated prefix/suffix. Piotrek > On Aug 31, 2017, at 7:12 PM, Raja.Aravapalli > wrote: > > > Hi, > > I have a flink application that is streaming data

Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Hi, I have a flink application that is streaming data into HDFS and I am using Bucketing Sink for that. And, I want to know if is it possible to rename the part files that is being created in the base hdfs directory. Right now I am using the below code for including the timestamp into