[
https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123627#comment-14123627
]
Colin Patrick McCabe commented on HDFS-6994:
--------------------------------------------
Hi Zhanwei, this is really interesting. As Wenwu mentioned, there's already a
branch where we're working on a native client. It would be nice if we could
integrate this with that work somehow. I'm not sure what form that should take.
Did you get a chance to read the design doc on HADOOP-10388? There are a few
issues important here before this can replace libhdfs. We need the ability to
fall back to the JNI code when necessary-- for example, in the case where HDFS
is using encryption, and we don't have native client support for that. But we
don't want a hard library dependency on libjvm.so-- it should be dynamically
loaded.
It's good that you are using the existing hdfs.h interface. getLastError seems
like it could be a good addition, as well... when using thread-local data for
the string.
The dependencies here are problematic. libxml2 is not fully thread-safe, and
it pulls in a lot of GNOME stuff we don't really want. The boost dependency
creates problems as well. For example, Impala depends on a certain version of
boost-- if this library pulls in a different version, bad things happen.
GnuTls is LGPL, which makes it difficult to ship. I would have to be -1 just
based on the dependencies alone...
We also have duplicated protobuf files in this patch. We should simply use the
protobuf files in the source tree already.
If I could summarize my first thoughts:
* get rid of boost, including all boost ifdefs
* don't worry about earlier RPC versions... we only need to support RPCv9 now
(same as Java client code policy in Hadoop)
* use libexpat or something instead of libxml2
This is good work overall and hopefully there is stuff we can use here.
> libhdfs3 - A native C/C++ HDFS client
> -------------------------------------
>
> Key: HDFS-6994
> URL: https://issues.apache.org/jira/browse/HDFS-6994
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client
> Reporter: Zhanwei Wang
> Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch
>
>
> Hi All
> I just got the permission to open source libhdfs3, which is a native C/C++
> HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.
> libhdfs3 provide the libhdfs style C interface and a C++ interface. Support
> both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos
> authentication.
> libhdfs3 is currently used by HAWQ of Pivotal
> I'd like to integrate libhdfs3 into HDFS source code to benefit others.
> You can find libhdfs3 code from github
> https://github.com/PivotalRD/libhdfs3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
