[
https://issues.apache.org/jira/browse/FLUME-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150457#comment-16150457
]
Denes Arvay commented on FLUME-3146:
------------------------------------
Hi [~jojochuang],
Thank you for your patch.
Isn't it possible to get rid of this reflection-magic alltogether?
Please correct me if I'm wrong but for me it seems that we can get the number
of replicas only if we have an {{HdfsDataOutputStream}} instance, in all the
other cases (i.e. {{MockFsDataOutputStream}}) it is not supported, so even with
the reflection the {{getNumCurrentReplicas}} method won't be found, thus the
{{getNumCurrentReplicas}} will return {{-1}}.
What do you think?
Thanks,
Denes
> Use public API HdfsDataOutputStream#getCurrentBlockReplication where
> applicable
> -------------------------------------------------------------------------------
>
> Key: FLUME-3146
> URL: https://issues.apache.org/jira/browse/FLUME-3146
> Project: Flume
> Issue Type: Bug
> Reporter: Wei-Chiu Chuang
> Attachments: FLUME-3146.001.patch
>
>
> AbstractHDFSWrite#reflectGetNumCurrentReplicas uses a private HDFS API, which
> causes it to emit an confusing error message:
> {noformat}
> 2017-08-08 22:01:10,743 INFO org.apache.flume.sink.hdfs.AbstractHDFSWriter:
> FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826
> not available; fsOut=org.apache.hadoop.crypto.CryptoOutputStream;
> err=java.lang.NoSuchMethodException:
> org.apache.hadoop.crypto.CryptoOutputStream.getNumCurrentReplicas()
> 2017-08-08 22:02:11,086 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter:
> Error while trying to hflushOrSync!
> {noformat}
> This error message is emitted even with the most recent version of HDFS that
> has HDFS-826 for sure.
> Turns out that the relevant code is not valid with HDFS encryption. Instead
> of using the current private (and deprecated)
> getNumCurrentReplicas#getNumCurrentReplicas() API, it should use
> HdfsDataOutputStream#getCurrentBlockReplication, which is a public API and
> which considers the case of HDFS encryption.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
