[
https://issues.apache.org/jira/browse/HDFS-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796556#comment-16796556
]
Hadoop QA commented on HDFS-14304:
----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
39s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-595/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/595 |
| JIRA Issue | HDFS-14304 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 85f868aca893 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1639071 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-595/4/testReport/ |
| Max. process+thread count | 410 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-595/4/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> High lock contention on hdfsHashMutex in libhdfs
> ------------------------------------------------
>
> Key: HDFS-14304
> URL: https://issues.apache.org/jira/browse/HDFS-14304
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> While doing some performance profiling of an application using libhdfs, we
> noticed a high amount of lock contention on the {{hdfsHashMutex}} defined in
> {{hadoop-hdfs-native-client/src/main/native/libhdfs/os/mutexes.h}}
> The issue is that every JNI method invocation done by {{hdfs.c}} goes through
> a helper method called {{invokeMethod}}. {{invokeMethod}} calls
> {{globalClassReference}} which acquires {{hdfsHashMutex}} while performing a
> lookup in a {{htable}} (a custom hash table that lives in {{libhdfs/common}})
> (the lock is acquired for both reads and writes). The hash table maps {{char
> *className}} to {{jclass}} objects, it seems the goal of the hash table is to
> avoid repeatedly creating {{jclass}} objects for each JNI call.
> For multi-threaded applications, this lock severely limits that rate at which
> Java methods can be invoked. pstacks show a lot of time being spent on
> {{hdfsHashMutex}}
> {code:java}
> #0 0x00007fba2dbc242d in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00007fba2dbbddcb in _L_lock_812 () from /lib64/libpthread.so.0
> #2 0x00007fba2dbbdc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x00000000027d8386 in mutexLock ()
> #4 0x00000000027d0e7b in globalClassReference ()
> #5 0x00000000027d1160 in invokeMethod ()
> #6 0x00000000027d4176 in readDirect ()
> #7 0x00000000027d4325 in hdfsRead ()
> {code}
> Same with {{perf report}}
> {code:java}
> + 63.36% 0.01% [k] system_call_fastpath
> + 61.60% 0.12% [k] sys_futex
> + 61.45% 0.13% [k] do_futex
> + 57.54% 0.49% [k] _raw_qspin_lock
> + 57.07% 0.01% [k] queued_spin_lock_slowpath
> + 55.47% 55.47% [k] native_queued_spin_lock_slowpath
> - 35.68% 0.00% [k] 0x6f6f6461682f6568
> - 0x6f6f6461682f6568
> - 30.55% __lll_lock_wait
> - 29.40% system_call_fastpath
> - 29.39% sys_futex
> - 29.35% do_futex
> - 29.27% futex_wait
> - 28.17% futex_wait_setup
> - 27.05% _raw_qspin_lock
> - 27.05% queued_spin_lock_slowpath
> 26.30% native_queued_spin_lock_slowpath
> + 0.67% ret_from_intr
> + 0.71% futex_wait_queue_me
> - 2.00% methodIdFromClass
> - 1.94% jni_GetMethodID
> - 1.71% get_method_id
> 0.96% SymbolTable::lookup_only
> - 1.61% invokeMethod
> - 0.62% jni_CallLongMethodV
> 0.52% jni_invoke_nonstatic
> 0.75% pthread_mutex_lock
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
