[
https://issues.apache.org/jira/browse/HDFS-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779549#comment-16779549
]
Sahil Takiar commented on HDFS-14083:
-------------------------------------
[~yzhangal] HDFS-14111 fixes this. It removes the log line
{{UnsupportedOperationException: Byte-buffer read unsupported by input}}
completely. Given the approach in HDFS-14111 I don't think the log line is ever
necessary. The {{StreamCapabilities}} interface in Hadoop captures whether or
not a stream supports the {{readDirect}} code path.
I suggest we close this JIRA in favor of HDFS-14111. The solution in this JIRA
is really just masking the problem rather than fixing it. As far I understand,
it decreases the frequency at which the logging occurs, whereas HDFS-14111
removes the logging completely. This has multiple benefits (1) it avoids
collecting a stack trace for each file open, and (2) simply decreasing the
frequency at which logging occurs still causes confusion for users, the log
indicates that some error occurred, whereas in reality lack of support for
{{readDirect}} is an expected limitation.
> libhdfs logs errors when opened FS doesn't support ByteBufferReadable
> ---------------------------------------------------------------------
>
> Key: HDFS-14083
> URL: https://issues.apache.org/jira/browse/HDFS-14083
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs, native
> Affects Versions: 3.0.3
> Reporter: Pranay Singh
> Assignee: Pranay Singh
> Priority: Minor
> Attachments: HADOOP-15928.001.patch, HADOOP-15928.002.patch,
> HDFS-14083.003.patch, HDFS-14083.004.patch, HDFS-14083.005.patch,
> HDFS-14083.006.patch, HDFS-14083.007.patch, HDFS-14083.008.patch,
> HDFS-14083.009.patch
>
>
> Problem:
> ------------
> There is excessive error logging when a file is opened by libhdfs
> (DFSClient/HDFS) in S3 environment, this issue is caused because buffered
> read is not supported in S3 environment, HADOOP-14603 "S3A input stream to
> support ByteBufferReadable"
> The following message is printed repeatedly in the error log/ to STDERR:
> {code}
> --------------------------------------------------------------------------------------------------
> UnsupportedOperationException: Byte-buffer read unsupported by input
> streamjava.lang.UnsupportedOperationException: Byte-buffer read unsupported
> by input stream
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150)
> {code}
> h3. Root cause
> After investigating the issue, it appears that the above exception is printed
> because
> when a file is opened via {{hdfsOpenFileImpl()}} calls {{readDirect()}} which
> is hitting this
> exception.
> h3. Fix:
> Since the hdfs client is not initiating the byte buffered read but is
> happening in a implicit manner, we should not be generating the error log
> during open of a file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
