[
https://issues.apache.org/jira/browse/HDFS-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437761#comment-13437761
]
Tsz Wo (Nicholas), SZE commented on HDFS-2656:
----------------------------------------------
Hi Jing,
Thanks for picking this up. I just have checked the read code.
Currently, hdfsOpenFile initializes some data structure, each hdfsRead creates
an individual URL connection (i.e. read n times have n URL connections) and
then hdfsCloseFile free the memory. hdfsOpenFile, hdfsRead and hdfsCloseFile
should be implemented with a single URL connection. I am not familiar with
libcurl, it seems that it could be done by
[curl_easy_pause|http://linux.die.net/man/3/curl_easy_pause]:
- hdfsOpenFile connects to the web server and then pause;
- hdfsRead let curl fill up the buffer and then pause;
- hdfsCloseFile closes the connection and free the memory.
Also, the are unnecessary memcpy in the current implementation. It creates
Response in launchCMD(..) and allocates memory in writefunc but not using the
buffer provided by the user. In hdfsRead, it copies the data from Response to
user buffer. I think we may be able to avoid the copying.
> Implement a pure c client based on webhdfs
> ------------------------------------------
>
> Key: HDFS-2656
> URL: https://issues.apache.org/jira/browse/HDFS-2656
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
> Reporter: Zhanwei.Wang
> Attachments: HDFS-2656.patch, HDFS-2656.unfinished.patch
>
>
> Currently, the implementation of libhdfs is based on JNI. The overhead of JVM
> seems a little big, and libhdfs can also not be used in the environment
> without hdfs.
> It seems a good idea to implement a pure c client by wrapping webhdfs. It
> also can be used to access different version of hdfs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
