[ https://issues.apache.org/jira/browse/HDFS-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765296#comment-16765296 ]
Todd Lipcon commented on HDFS-14083: ------------------------------------ {quote} if (errno != 0) { file->flags &= ~HDFS_FILE_SUPPORTS_DIRECT_READ; } {quote} this doesn't seem quite right, since there isn't a guarantee that 'readDirect' sets errno to 0 in the case of success. It might just pick up some previously-existing value there. Also, the handling of those static variables seems thread-unsafe. It might work OK since the values are small enough to be naturally atomic on x86 but it seems like we might be better off addressing the root of this issue a better way instead of throttling the error logging? > libhdfs logs errors when opened FS doesn't support ByteBufferReadable > --------------------------------------------------------------------- > > Key: HDFS-14083 > URL: https://issues.apache.org/jira/browse/HDFS-14083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: native, test > Affects Versions: 3.0.3 > Reporter: Pranay Singh > Assignee: Pranay Singh > Priority: Minor > Attachments: HADOOP-15928.001.patch, HADOOP-15928.002.patch, > HDFS-14083.003.patch, HDFS-14083.004.patch, HDFS-14083.005.patch, > HDFS-14083.006.patch, HDFS-14083.007.patch, HDFS-14083.008.patch, > HDFS-14083.009.patch > > > Problem: > ------------ > There is excessive error logging when a file is opened by libhdfs > (DFSClient/HDFS) in S3 environment, this issue is caused because buffered > read is not supported in S3 environment, HADOOP-14603 "S3A input stream to > support ByteBufferReadable" > The following message is printed repeatedly in the error log/ to STDERR: > {code} > -------------------------------------------------------------------------------------------------- > UnsupportedOperationException: Byte-buffer read unsupported by input > streamjava.lang.UnsupportedOperationException: Byte-buffer read unsupported > by input stream > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150) > {code} > h3. Root cause > After investigating the issue, it appears that the above exception is printed > because > when a file is opened via {{hdfsOpenFileImpl()}} calls {{readDirect()}} which > is hitting this > exception. > h3. Fix: > Since the hdfs client is not initiating the byte buffered read but is > happening in a implicit manner, we should not be generating the error log > during open of a file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org