Re: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing
Thanks Guowei. Good to know it may have been fixed in another way. I'll have a try. BTW, which version is this kind of sink supported from? Does it also work for batch mode? On Mon, Apr 25, 2022 at 2:42 PM Guowei Ma wrote: > Hi > > Afaik the commit files action happens at the committer operator instead of > the JM size after the new sink api [1]. > > It means this would not happen if you use the new `FlinkSink`.[2] > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > > [2] > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java > > Best, > Guowei > > > On Sun, Apr 24, 2022 at 11:39 AM Yi Tang wrote: > >> >> >> -- Forwarded message - >> From: Yi Tang >> Date: Sun, Apr 24, 2022 at 11:29 AM >> Subject: Write to Aliyun OSS via FileSystem connector hang Job Master on >> Finishing >> To: >> >> >> Hi team; >> >> I'm trying to write to aliyun oss via FileSystem connector. The job >> master always hangs on finishing. >> >> Looks like it is because the FileSystem connector commits the files by >> #finalizeGlobal while the Job is finishing, which includes some rename >> operations. However, the aliyun oss FileSystem renames files by copying, >> which seems expensive. >> >> Any suggestions about this scenario? >> >> Thanks and regards. >> >>
Re: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing
Hi Afaik the commit files action happens at the committer operator instead of the JM size after the new sink api [1]. It means this would not happen if you use the new `FlinkSink`.[2] [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API [2] https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java Best, Guowei On Sun, Apr 24, 2022 at 11:39 AM Yi Tang wrote: > > > -- Forwarded message - > From: Yi Tang > Date: Sun, Apr 24, 2022 at 11:29 AM > Subject: Write to Aliyun OSS via FileSystem connector hang Job Master on > Finishing > To: > > > Hi team; > > I'm trying to write to aliyun oss via FileSystem connector. The job master > always hangs on finishing. > > Looks like it is because the FileSystem connector commits the files by > #finalizeGlobal while the Job is finishing, which includes some rename > operations. However, the aliyun oss FileSystem renames files by copying, > which seems expensive. > > Any suggestions about this scenario? > > Thanks and regards. > >
Fwd: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing
-- Forwarded message - From: Yi Tang Date: Sun, Apr 24, 2022 at 11:29 AM Subject: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing To: Hi team; I'm trying to write to aliyun oss via FileSystem connector. The job master always hangs on finishing. Looks like it is because the FileSystem connector commits the files by #finalizeGlobal while the Job is finishing, which includes some rename operations. However, the aliyun oss FileSystem renames files by copying, which seems expensive. Any suggestions about this scenario? Thanks and regards.
