On 13.10.20 11:18, David Anderson wrote:
I think the pertinent question is whether there are interesting cases where the BucketingSink is still a better choice. One case I'm not sure about is the situation described in docs for the StreamingFileSink under Important Note 2 [1]:... upon normal termination of a job, the last in-progress files will not be transitioned to the “finished” state. I know this confuses and frustrates users, but I don't know if the BucketingSink has any advantages in this regard.
The BucketingSink suffers from the same problem. It's caused by the fact that we don't do a "final" checkpoint before shutting down a pipeline. We're trying to resolve that with FLIP-147 [1].
[1] https://cwiki.apache.org/confluence/x/mw-ZCQ
