Re: expr-test stuck in getJNIEnv
The upstream bug is posted below. I've requested a backport to the HDFS version Impala depends on, but it hasn't happened yet. I'll make another push for it. https://issues.apache.org/jira/browse/HDFS-11851 On Sun, Sep 10, 2017 at 4:01 PM, Jim Apple wrote: > It was . bin/set-classpath.sh. Forgot about that one. Thanks, Henry! > > On Sun, Sep 10, 2017 at 3:31 PM, Henry Robinson wrote: > > I've seen this deadlock before although not in expr-test. I can't > remember > > exactly how I cleared it but I believe it was either: > > > > 1. make fe && . bin/set-classpath.sh > > 2. bin/create-test-configuration.sh > > > > Sailesh knows about the upstream HDFS bug which I think has been fixed > but > > not incorporated into Impala's dependencies. > > > > On Sun, Sep 10, 2017 at 1:42 PM Jim Apple wrote: > > > >> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full > stack > >> trace: > >> > >> #0 __lll_lock_wait () at > >> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 > >> #1 0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0 > >> ) at ../nptl/pthread_mutex_lock.c:80 > >> #2 0x02cf79f6 in mutexLock (m=) at > >> > >> /data/2/jenkins/workspace/impala-hadoop-dependency/ > hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ > libhdfs/os/posix/mutexes.c:28 > >> #3 0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0, > >> stackTrace=0x0) at > >> > >> /data/2/jenkins/workspace/impala-hadoop-dependency/ > hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ > libhdfs/jni_helper.c:581 > >> #4 0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8, > >> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994 > >> "loadFileSystems", ap=0x7fff9da0) > >> at > >> /data/2/jenkins/workspace/impala-hadoop-dependency/ > hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ > libhdfs/exception.c:183 > >> #5 0x02cf81dd in printExceptionAndFree (env=, > >> exc=, noPrintFlags=, fmt= >> out>) > >> at > >> /data/2/jenkins/workspace/impala-hadoop-dependency/ > hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ > libhdfs/exception.c:213 > >> #6 0x02cf0faf in getGlobalJNIEnv () at > >> > >> /data/2/jenkins/workspace/impala-hadoop-dependency/ > hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ > libhdfs/jni_helper.c:463 > >> #7 getJNIEnv () at > >> > >> /data/2/jenkins/workspace/impala-hadoop-dependency/ > hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ > libhdfs/jni_helper.c:528 > >> #8 0x01a0116a in impala::JniUtil::Init () at > >> be/src/util/jni-util.cc:105 > >> #9 0x014fd881 in impala::InitCommonRuntime (argc=1, > >> argv=0x7fffa628, init_jvm=true, > >> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236 > >> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at > >> be/src/exprs/expr-test.cc:7420 > >> > >> I've tried git fetch, bin/clean.sh, running with the minicluster on, > >> running with the minicluster off, running with the impala cluster on, > >> running with it off, running in release mode, debug mode, in gdb, and > >> out of gdb. > >> > >> Has anyone else seen this and escaped from its clutches? > >> >
Re: expr-test stuck in getJNIEnv
It was . bin/set-classpath.sh. Forgot about that one. Thanks, Henry! On Sun, Sep 10, 2017 at 3:31 PM, Henry Robinson wrote: > I've seen this deadlock before although not in expr-test. I can't remember > exactly how I cleared it but I believe it was either: > > 1. make fe && . bin/set-classpath.sh > 2. bin/create-test-configuration.sh > > Sailesh knows about the upstream HDFS bug which I think has been fixed but > not incorporated into Impala's dependencies. > > On Sun, Sep 10, 2017 at 1:42 PM Jim Apple wrote: > >> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack >> trace: >> >> #0 __lll_lock_wait () at >> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 >> #1 0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0 >> ) at ../nptl/pthread_mutex_lock.c:80 >> #2 0x02cf79f6 in mutexLock (m=) at >> >> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28 >> #3 0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0, >> stackTrace=0x0) at >> >> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581 >> #4 0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8, >> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994 >> "loadFileSystems", ap=0x7fff9da0) >> at >> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183 >> #5 0x02cf81dd in printExceptionAndFree (env=, >> exc=, noPrintFlags=, fmt=> out>) >> at >> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213 >> #6 0x02cf0faf in getGlobalJNIEnv () at >> >> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463 >> #7 getJNIEnv () at >> >> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528 >> #8 0x01a0116a in impala::JniUtil::Init () at >> be/src/util/jni-util.cc:105 >> #9 0x014fd881 in impala::InitCommonRuntime (argc=1, >> argv=0x7fffa628, init_jvm=true, >> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236 >> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at >> be/src/exprs/expr-test.cc:7420 >> >> I've tried git fetch, bin/clean.sh, running with the minicluster on, >> running with the minicluster off, running with the impala cluster on, >> running with it off, running in release mode, debug mode, in gdb, and >> out of gdb. >> >> Has anyone else seen this and escaped from its clutches? >>
Re: expr-test stuck in getJNIEnv
I've seen this deadlock before although not in expr-test. I can't remember exactly how I cleared it but I believe it was either: 1. make fe && . bin/set-classpath.sh 2. bin/create-test-configuration.sh Sailesh knows about the upstream HDFS bug which I think has been fixed but not incorporated into Impala's dependencies. On Sun, Sep 10, 2017 at 1:42 PM Jim Apple wrote: > When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack > trace: > > #0 __lll_lock_wait () at > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 > #1 0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0 > ) at ../nptl/pthread_mutex_lock.c:80 > #2 0x02cf79f6 in mutexLock (m=) at > > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28 > #3 0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0, > stackTrace=0x0) at > > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581 > #4 0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8, > exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994 > "loadFileSystems", ap=0x7fff9da0) > at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183 > #5 0x02cf81dd in printExceptionAndFree (env=, > exc=, noPrintFlags=, fmt= out>) > at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213 > #6 0x02cf0faf in getGlobalJNIEnv () at > > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463 > #7 getJNIEnv () at > > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528 > #8 0x01a0116a in impala::JniUtil::Init () at > be/src/util/jni-util.cc:105 > #9 0x014fd881 in impala::InitCommonRuntime (argc=1, > argv=0x7fffa628, init_jvm=true, > test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236 > #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at > be/src/exprs/expr-test.cc:7420 > > I've tried git fetch, bin/clean.sh, running with the minicluster on, > running with the minicluster off, running with the impala cluster on, > running with it off, running in release mode, debug mode, in gdb, and > out of gdb. > > Has anyone else seen this and escaped from its clutches? >
expr-test stuck in getJNIEnv
When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack trace: #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x02cf79f6 in mutexLock (m=) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28 #3 0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0, stackTrace=0x0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581 #4 0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8, exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994 "loadFileSystems", ap=0x7fff9da0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183 #5 0x02cf81dd in printExceptionAndFree (env=, exc=, noPrintFlags=, fmt=) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213 #6 0x02cf0faf in getGlobalJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463 #7 getJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528 #8 0x01a0116a in impala::JniUtil::Init () at be/src/util/jni-util.cc:105 #9 0x014fd881 in impala::InitCommonRuntime (argc=1, argv=0x7fffa628, init_jvm=true, test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236 #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at be/src/exprs/expr-test.cc:7420 I've tried git fetch, bin/clean.sh, running with the minicluster on, running with the minicluster off, running with the impala cluster on, running with it off, running in release mode, debug mode, in gdb, and out of gdb. Has anyone else seen this and escaped from its clutches?