Hi all, I noticed that it isn’t possible to specify empty inUseSuffix for the HDFS sink. As a result, we can't avoid a redundant rename operation for files in s3. But afaik, s3 native filesystem keeps a temporary file on the local fs of the machine. So, we anyway copy on s3 the final data, right?
At the same time the rename operation for s3 is copy + delete, which is kinda useless if we already have a final data in the "temporary" file. The question is: how can I avoid the rename operation in a HDFS sink? Thanks in advance! - Dmytro Popovych
