[ https://issues.apache.org/jira/browse/HADOOP-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-16107: ------------------------------------ Status: Open (was: Patch Available) > FilterFileSystem doesn't wrap all create() or new builder calls; may skip CRC > logic > ----------------------------------------------------------------------------------- > > Key: HADOOP-16107 > URL: https://issues.apache.org/jira/browse/HADOOP-16107 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 3.0.3, 3.3.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Blocker > Attachments: HADOOP-16107-001.patch > > > LocalFS is a subclass of filterFS, but overrides create and open so that > checksums are created and read. > MAPREDUCE-7184 has thrown up that the new builder openFile() call is being > forwarded to the innerFS without CRC checking. Reviewing/fixing that has > shown that some of the create methods aren't being correctly wrapped, so not > generating CRCs > * createFile() builder > The following create calls > {code} > public FSDataOutputStream createNonRecursive(final Path f, > final FsPermission permission, > final EnumSet<CreateFlag> flags, > final int bufferSize, > final short replication, > final long blockSize, > final Progressable progress) throws IOException; > public FSDataOutputStream create(final Path f, > final FsPermission permission, > final EnumSet<CreateFlag> flags, > final int bufferSize, > final short replication, > final long blockSize, > final Progressable progress, > final Options.ChecksumOpt checksumOpt) throws IOException { > return super.create(f, permission, flags, bufferSize, replication, > blockSize, progress, checksumOpt); > } > {code} > This means that applications using these methods, directly or indirectly to > create files aren't actually generating checksums. > Fix: implement these methods & relay to local create calls, not to the inner > FS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org