[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032247#comment-14032247 ] Wenwu Peng commented on HADOOP-10640: - bq.Wenwu, Binglin, are you testing the latest version (v3) ? It includes this fix: Hi Colin, Still get OutOfMemoryException in latest version(v4) ,Could anyone take a look? OS: Ubuntu 12.04.1 hdfsBuilderConnect: ndfs failed to connect: org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctx (error 12) org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctxerror: did not expect '0': '/home/hary/hadoop-common/hadoop-native-core/src/main/native/test/fs/test_libhdfs_meta_ops.c at line 60' Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: HADOOP-10388 Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch, HADOOP-10640-pnative.004.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029690#comment-14029690 ] Abraham Elmahrek commented on HADOOP-10640: --- Thanks for addressing my comments Colin. +1. Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch, HADOOP-10640-pnative.004.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026747#comment-14026747
]
Abraham Elmahrek commented on HADOOP-10640:
---
Awesome stuff Colin. Just a few comments:
* Do we need to call free in hadoop_err_prepend.c for asprintf error cases?
Docs say no memory is allocated in this case.
{code}
if (asprintf(nmsg, %s: %s, prepend_str, err-msg) 0) {
free(prepend_str);
return (struct hadoop_err*)err;
}
{code}
* The hash table implementation has an unbounded while loop. Though, it will
probably never happen since we guarantee there will always be an open spot,
would we add a terminal case to it?
{code}
static void htable_insert_internal(struct htable_pair *nelem,
uint32_t capacity, htable_hash_fn_t hash_fun, void *key,
void *val)
{
uint32_t i;
i = hash_fun(key, capacity);
while (1) {
if (!nelem[i].key) {
nelem[i].key = key;
nelem[i].val = val;
return;
}
i++;
if (i == capacity) {
i = 0;
}
}
}
{code}
* Should the above hash table be modified to allow custom hash functions in the
future? Modifications would include ensuring the hash function was within
bounds, providing an interface, etc.
* The config object seems to be using the builder pattern. Wouldn't it make
sense to just create a configuration object and provide 'set' and 'get'
functions? Unless the configuration object is immutable?
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch,
HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026841#comment-14026841
]
Colin Patrick McCabe commented on HADOOP-10640:
---
bq. Do we need to call free in hadoop_err_prepend.c for asprintf error cases?
Docs say no memory is allocated in this case.
The string being freed here was allocated further up in the function. It's not
allocated by the (failed) asprintf.
bq. The hash table implementation has an unbounded while loop. Though, it will
probably never happen since we guarantee there will always be an open spot,
would we add a terminal case to it?
The hash table is never more than half full. Check this code in {{htable_put}}:
{code}
+// Re-hash if we have used more than half of the hash table
+nused = htable-used + 1;
+if (nused = (htable-capacity / 2)) {
+ret = htable_realloc(htable, htable-capacity * 2);
+if (ret)
+return ret;
+}
+htable_insert_internal(htable-elem, htable-capacity,
+htable-hash_fun, key, val);
{code}
I will add a comment to {{htable_insert_internal}} making this invariant clear.
bq. Should the above hash table be modified to allow custom hash functions in
the future? Modifications would include ensuring the hash function was within
bounds, providing an interface, etc.
Already done :)
{code}
+struct htable *htable_alloc(uint32_t size,
+htable_hash_fn_t hash_fun, htable_eq_fn_t eq_fun)
{code}
You can supply your own hash function as the {{hash_fun}} argument.
bq. The config object seems to be using the builder pattern. Wouldn't it make
sense to just create a configuration object and provide 'set' and 'get'
functions? Unless the configuration object is immutable?
The configuration object is immutable once created. I wanted to avoid
multithreading problems with get and set... we've had a lot of those with
{{Configuration}} in Hadoop. This also simplifies the C code, since we can
simply use strings from inside the {{hconf}} object without worrying about
whether someone is going to {{free}} them while we're using them. This means,
for example, that we don't need to copy them inside {{hconf_get}}. All the
strings get freed at the end, when the {{hconf}} is freed. I'll add a comment
that hconf is immutable.
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch,
HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025478#comment-14025478
]
Colin Patrick McCabe commented on HADOOP-10640:
---
bq. sizeof(struct hrpc_proxy) always larger than RPC_PROXY_USERDATA_MAX?
Wenwu, Binglin, are you testing the latest version (v3) ? It includes this fix:
{code}
diff --git hadoop-native-core/src/main/native/rpc/proxy.c
hadoop-native-core/src/main/native/rpc/proxy.c
index 9d69788..faa3571 100644
--- hadoop-native-core/src/main/native/rpc/proxy.c
+++ hadoop-native-core/src/main/native/rpc/proxy.c
@@ -92,7 +92,7 @@ void *hrpc_proxy_alloc_userdata(struct hrpc_proxy *proxy,
size_t size)
struct hrpc_sync_ctx *hrpc_proxy_alloc_sync_ctx(struct hrpc_proxy *proxy)
{
struct hrpc_sync_ctx *ctx =
-hrpc_proxy_alloc_userdata(proxy, sizeof(struct hrpc_proxy));
+hrpc_proxy_alloc_userdata(proxy, sizeof(*ctx));
if (!ctx) {
return NULL;
}
{code}
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch,
HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021541#comment-14021541 ] Wenwu Peng commented on HADOOP-10640: - run command NAMENODE_URI=hdfs://localhost:8020 ./test_libhdfs_meta_ops and hit this error hdfsBuilderConnect: ndfs failed to connect: org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctx (error 12) org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctxerror: did not expect '0': '/root/hadoop-common/hadoop-native-core/src/main/native/test/fs/test_libhdfs_meta_ops.c at line 60' Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021649#comment-14021649
]
Binglin Chang commented on HADOOP-10640:
sizeof(struct hrpc_proxy) always larger than RPC_PROXY_USERDATA_MAX?
{code}
void *hrpc_proxy_alloc_userdata(struct hrpc_proxy *proxy, size_t size)
{
if (size RPC_PROXY_USERDATA_MAX) {
return NULL;
}
return proxy-userdata;
}
struct hrpc_sync_ctx *hrpc_proxy_alloc_sync_ctx(struct hrpc_proxy *proxy)
{
struct hrpc_sync_ctx *ctx =
hrpc_proxy_alloc_userdata(proxy, sizeof(struct hrpc_proxy));
if (!ctx) {
return NULL;
}
if (uv_sem_init(ctx-sem, 0)) {
return NULL;
}
memset(ctx, 0, sizeof(ctx));
return ctx;
}
{code}
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch,
HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018861#comment-14018861
]
Binglin Chang commented on HADOOP-10640:
In hdfs.h:
{code}
#if defined(unix) || defined(__MACH__)
{code}
bq. I don't really like the typedefs. They make it hard to forward-declare
structures in header files.
I see some methods in ndfs/jnifs use typedef, some use structs, as long as they
are uniform in impls.
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch,
HADOOP-10640-pnative.002.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019090#comment-14019090
]
Colin Patrick McCabe commented on HADOOP-10640:
---
bq. /root/hadoop-common/hadoop-native-core/src/main/native/ndfs/ndfs.c:315:
warning: ‘defaults.blocksize’ may be used uninitialized in this function
It's a spurious warning, since {{ndfs_get_server_defaults}} will either return
an error or set {{defaults.blocksize}}. But I'll add a {{memset}} to avoid it.
This will also make things cleaner if we add more elements to the defaults
structure in the future.
bq. #if defined(unix) || defined(__MACH__)
Hmm. I thought that {{unix}} would be defined on MacOS. I guess not. I will
add this-- thanks.
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch,
HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018120#comment-14018120
]
Colin Patrick McCabe commented on HADOOP-10640:
---
bq. CMakeLists.txt: #add_subdirectory(fs) fs/CMakeLists.txt redundant?
Oops, this was from an earlier version. Removed.
bq. CMakeList.txt: -fvisibility=hidden macosx also support this
According to the CMake documentation, UNIX is TRUE on all UNIX-like OS's,
including Apple OS X and CygWin. So my if statement should activate on
MacOS... let me know if it doesn't for you.
bq. fs/fs.h:136 use hdfsFile/hdfsFs instead of struct hdfsFile_internal
*/struct hdfs_internal ?
I don't really like the typedefs. They make it hard to forward-declare
structures in header files.
bq. config.h.cmake HCONF_XML_TEST_PATH we can set CLASSPATH env in tests, it
better than static config macro
I wanted make test to keep working without any special effort on behalf of
the user running make. If you can think of a way to do this without the
{{HCONF_XML_TEST_PATH}} macro, let me know.
bq. should add ${JNI_INCLUDE_DIRS} in include_directories:
added, thanks.
bq.
/Users/decster/projects/hadoop-trunk/hadoop-native-core/ndfs/ndfs.c:1055:14:
warning: incompatible pointer types initializing 'int (struct hdfs_internal *,
const char *, int64_t, int64_t)' with int64_t
Ah, you reminded me. I need to multiply these numbers by 1000 to take into
account the fact that the libhdfs API only does second (not millisecond)
precision (unfortunately).
In general, I would like to get rid of {{tTime}} at some point. It doesn't
really do anything useful. It makes time 32 bits on 32-bit systems, and I
don't see why we would want that. It should just be 64 bits everywhere, hence
{{int64_t}}. This matches up with what it is in Java... a Java long is always
64 bits and signed. So I use {{int64_t}} for all the internal functions and
only deal with {{tTime}} in {{fs.c}} and {{hdfs.h}}.
bq. /Users/decster/projects/hadoop-trunk/hadoop-native-core/fs/common.c:39:36:
warning: 'memset' call operates on objects of type 'hdfsFileInfo' (aka 'struct
file_info') while the size is based on a
fixed, thanks.
bq. /Users/decster/projects/hadoop-trunk/hadoop-native-core/rpc/proxy.c:102:27:
warning: 'memset' call operates on objects of type 'struct hrpc_sync_ctx' while
the size is based on a different type
fixed
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018443#comment-14018443 ] Wenwu Peng commented on HADOOP-10640: - Thanks Colin big effort. could you help take a look the warning? /root/hadoop-common/hadoop-native-core/src/main/native/ndfs/ndfs.c:315: warning: ‘defaults.blocksize’ may be used uninitialized in this function Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016318#comment-14016318
]
Binglin Chang commented on HADOOP-10640:
Thanks for the patch Colin. Have not finished reviewing, some comments:
CMakeLists.txt: #add_subdirectory(fs) fs/CMakeLists.txt redundant?
CMakeLists.txt: #include(Libhdfs.cmake) do not have this file
CMakeList.txt: -fvisibility=hidden macosx also support this(if
(${CMAKE_SYSTEM_NAME} MATCHES Darwin) to detected), you can add this or I can
add it later.
fs/fs.h:136 use hdfsFile/hdfsFs instead of struct hdfsFile_internal */struct
hdfs_internal ?
config.h.cmake HCONF_XML_TEST_PATH we can set CLASSPATH env in tests, it better
than static config macro
when compiling(looks like clang can find more code bugs than gcc...):
should add ${JNI_INCLUDE_DIRS} in include_directories:
In file included from
/Users/decster/projects/hadoop-trunk/hadoop-native-core/test/native_mini_dfs.c:21:
/Users/decster/projects/hadoop-trunk/hadoop-native-core/jni/exception.h:37:10:
fatal error: 'jni.h' file not found
#include jni.h
should unified to tTime:
/Users/decster/projects/hadoop-trunk/hadoop-native-core/ndfs/ndfs.c:1055:14:
warning: incompatible pointer types initializing 'int (*)(struct hdfs_internal
*, const char *, int64_t, int64_t)' with
an expression of type 'int (hdfsFS, const char *, tTime, tTime)'
[-Wincompatible-pointer-types]
.utime = ndfs_utime,
^~
wrong memset usage:
/Users/decster/projects/hadoop-trunk/hadoop-native-core/fs/common.c:39:36:
warning: 'memset' call operates on objects of type 'hdfsFileInfo' (aka 'struct
file_info') while the size is based on a
different type 'hdfsFileInfo *' (aka 'struct file_info *')
[-Wsizeof-pointer-memaccess]
memset(hdfsFileInfo, 0, sizeof(hdfsFileInfo));
^~~~
/Users/decster/projects/hadoop-trunk/hadoop-native-core/rpc/proxy.c:102:27:
warning: 'memset' call operates on objects of type 'struct hrpc_sync_ctx' while
the size is based on a different type
'struct hrpc_sync_ctx *' [-Wsizeof-pointer-memaccess]
memset(ctx, 0, sizeof(ctx));
~~~^~~
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016319#comment-14016319
]
Binglin Chang commented on HADOOP-10640:
bad format... re submit comments.
{noformat}
CMakeLists.txt: #add_subdirectory(fs) fs/CMakeLists.txt redundant?
CMakeLists.txt: #include(Libhdfs.cmake) do not have this file
CMakeList.txt: -fvisibility=hidden macosx also support this(if
(${CMAKE_SYSTEM_NAME} MATCHES Darwin) to detected), you can add this or I can
add it later.
fs/fs.h:136 use hdfsFile/hdfsFs instead of struct hdfsFile_internal */struct
hdfs_internal ?
config.h.cmake HCONF_XML_TEST_PATH we can set CLASSPATH env in tests, it better
than static config macro
{noformat}
when compiling(looks like clang can find more code bugs than gcc...):
{noformat}
should add ${JNI_INCLUDE_DIRS} in include_directories:
In file included from
/Users/decster/projects/hadoop-trunk/hadoop-native-core/test/native_mini_dfs.c:21:
/Users/decster/projects/hadoop-trunk/hadoop-native-core/jni/exception.h:37:10:
fatal error: 'jni.h' file not found
#include jni.h
{noformat}
should unified to tTime:
{noformat}
/Users/decster/projects/hadoop-trunk/hadoop-native-core/ndfs/ndfs.c:1055:14:
warning: incompatible pointer types initializing 'int (*)(struct hdfs_internal
*, const char *, int64_t, int64_t)' with
an expression of type 'int (hdfsFS, const char *, tTime, tTime)'
[-Wincompatible-pointer-types]
.utime = ndfs_utime,
^~
{noformat}
wrong memset usage:
{noformat}
/Users/decster/projects/hadoop-trunk/hadoop-native-core/fs/common.c:39:36:
warning: 'memset' call operates on objects of type 'hdfsFileInfo' (aka 'struct
file_info') while the size is based on a
different type 'hdfsFileInfo *' (aka 'struct file_info *')
[-Wsizeof-pointer-memaccess]
memset(hdfsFileInfo, 0, sizeof(hdfsFileInfo));
^~~~
/Users/decster/projects/hadoop-trunk/hadoop-native-core/rpc/proxy.c:102:27:
warning: 'memset' call operates on objects of type 'struct hrpc_sync_ctx' while
the size is based on a different type
'struct hrpc_sync_ctx *' [-Wsizeof-pointer-memaccess]
memset(ctx, 0, sizeof(ctx));
~~~^~~
{noformat}
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HADOOP-10640-pnative.001.patch
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[
https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012839#comment-14012839
]
Colin Patrick McCabe commented on HADOOP-10640:
---
With this patch, libhdfs.so continues to implement the same API as before. At
runtime, we select which native filesystem implementation to use based on by
looking at {{scheme.native.handler}}. So, if the scheme of the URI we're
handling is HDFS, we'd look at {{hdfs.native.handler}}. If there is no
scheme-specific handler, we fall back to {{default.native.handler}}.
There are two libhdfs native filesystems available at the moment. One is
jniFS, the existing JNI-based filesystem. The other is NDFS, the new native
filesystem that we're developing. As mentioned earlier, jniFS has the
advantage of working with things like s3 or HDFS encryption, whereas NDFS has
the advantage that it doesn't require JNI, is more efficient, etc. The goal
here is to have a {{libhdfs.so}} that we can drop into existing installations
and have everything just work.
The libhdfs.so library doesn't depend on JNI... it can be installed in cases
where JNI is not present. The reason is because it uses {{dlopen}} (by means
of the libuv shims) to open the {{libjvm.so}} library as-needed. So clients
who don't want to install or use JNI don't have to.
The build changed a little bit to have a {{config.h.cmake}} file which gets
transformed into {{config.h}} based on which features are present at
compile-time. This is how the existing libhdfs build works (we need it to
detect some platform-specific features.) I also hide all library symbols
except for the public symbols, so that our library internals are not visible to
the outside world, using {{fvisibility=hidden}}.
source overview:
* common/hadoop_err: added the {{hadoop_err_prepend}}, {{hadoop_err_copy}}, and
{{terror}} utility functions.
* common/hconf: this is a native implementation of the Java Configuration class
which reads XML files. It uses libexpat.
* common/htable: a hash table used in a few places.
* common/net: added {{net_ipv4_name_and_port}}, {{get_first_ipv4_addr}} utility
functions
* common/string: added the {{strdupto}} utility function plus a unit test.
* common/uri: parses HDFS-style URIs into scheme, authority, port, etc. using
the uriparser library.
* fs/fs.c: the entry points for all libhdfs functions. Contains a vtable used
to call the appropriate implementation function.
* fs/hdfs.h: the public API of libhdfs. This is just a rename.
* jni/: all the stuff in this directory was moved from the previous libhdfs
location.
* net/proxy, net/reactor: fixed a bug in the synchronous RPC code
* test/: I created a test directory paralleling the source directory. Moved
test.h to this directory.
Running {{NAMENODE_URI=hdfs://localhost:6000 ./test_libhdfs_meta_ops}} will
give an idea of how this operates.
Implement Namenode RPCs in HDFS native client
-
Key: HADOOP-10640
URL: https://issues.apache.org/jira/browse/HADOOP-10640
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: HADOOP-10388
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Implement the parts of libhdfs that just involve making RPCs to the Namenode,
such as mkdir, rename, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012863#comment-14012863 ] Colin Patrick McCabe commented on HADOOP-10640: --- I posted a reviewboard at https://reviews.apache.org/r/22046/, in case it's easier to review it in that format Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HADOOP-10640-pnative.001.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
