Hi Afaik the commit files action happens at the committer operator instead of the JM size after the new sink api [1].
It means this would not happen if you use the new `FlinkSink`.[2] [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API [2] https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java Best, Guowei On Sun, Apr 24, 2022 at 11:39 AM Yi Tang <ssnailt...@gmail.com> wrote: > > > ---------- Forwarded message --------- > From: Yi Tang <ssnailt...@gmail.com> > Date: Sun, Apr 24, 2022 at 11:29 AM > Subject: Write to Aliyun OSS via FileSystem connector hang Job Master on > Finishing > To: <user@flink.apache.org> > > > Hi team; > > I'm trying to write to aliyun oss via FileSystem connector. The job master > always hangs on finishing. > > Looks like it is because the FileSystem connector commits the files by > #finalizeGlobal while the Job is finishing, which includes some rename > operations. However, the aliyun oss FileSystem renames files by copying, > which seems expensive. > > Any suggestions about this scenario? > > Thanks and regards. > >
