todd created FLINK-23725: ---------------------------- Summary: HadoopFsCommitter, Prompt for file rename failure Key: FLINK-23725 URL: https://issues.apache.org/jira/browse/FLINK-23725 Project: Flink Issue Type: Bug Components: Connectors / FileSystem, Connectors / Hadoop Compatibility Affects Versions: 1.12.1, 1.11.1 Reporter: todd
When the HDFS file is written, if the part file exists, only false will be returned if the duplicate name fails.Whether to throw an exception that already exists in the part, or print related logs. ``` org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit public void commit() throws IOException { final Path src = recoverable.tempFile(); final Path dest = recoverable.targetFile(); final long expectedLength = recoverable.offset(); final FileStatus srcStatus; try { srcStatus = fs.getFileStatus(src); } catch (IOException e) { throw new IOException("Cannot clean commit: Staging file does not exist."); } if (srcStatus.getLen() != expectedLength) { // something was done to this file since the committer was created. // this is not the "clean" case throw new IOException("Cannot clean commit: File has trailing junk data."); } try { // return false or ture fs.rename(src, dest); } catch (IOException e) { throw new IOException( "Committing file by rename failed: " + src + " to " + dest, e); } } -- This message was sent by Atlassian Jira (v8.3.4#803005)