[
https://issues.apache.org/jira/browse/HDFS-7887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348008#comment-14348008
]
Haohui Mai commented on HDFS-7887:
----------------------------------
Given the popularity of integrating Hadoop with other native applications,
there are several attempts to implement a native RPC library. The most recent
attempts are HADOOP-10389 and HDFS-7013. The jira proposes to combine the
previous efforts and creates a unified libraries. The existing implementations
in HADOOP-10389 and HDFS-7013 have several drawbacks:
* The implementation is tightly coupled with the native HDFS client, making it
unavailable for YARN.
* Both HADOOP-10389 and HDFS-7013 only provide synchronous APIs. It fails to be
a building block for an asynchronous higher-level client.
* HDFS-7013 uses C++ exceptions extensively throughout the code. The community
has expressed the concerns on C++ exceptions and would like to get them removed
before the code is merged.
This jira proposes to implement the following components:
* A compiler that generates stubs from protobuf definitions, which can be taken
from HADOOP-10389.
* An asynchronous runtime for the Hadoop RPC
* Supports for SASL, wire encryption, and Kerberos authentication, which can be
taken from HDFS-7013.
* Unit tests, many of which can be taken from HADOOP-10389 and HDFS-7013.
> Asynchronous native RPC v9 client
> ---------------------------------
>
> Key: HDFS-7887
> URL: https://issues.apache.org/jira/browse/HDFS-7887
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Haohui Mai
> Assignee: Haohui Mai
>
> There are more and more integration happening between Hadoop and applications
> that are implemented using languages other than Java.
> To access Hadoop, applications either have to go through JNI (e.g. libhdfs),
> or to reverse engineer the Hadoop RPC protocol. (e.g. snakebite).
> Unfortunately, neither of them are satisfactory:
> * Integrating with JNI requires running a JVM inside the application. Some
> applications (e.g., real-time processing, MPP database) does not want the
> footprints and GC behavior of the JVM.
> * The Hadoop RPC protocol has a rich feature set including wire encryption,
> SASL, Kerberos authentication. Many 3rd-party implementations can fully cover
> the feature sets thus they might work in limited environment.
> This jira is to propose implementing an Hadoop RPC library in C++ that
> provides a common ground to implement higher-level native client for HDFS,
> YARN, and MapReduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
