Re: expr-test stuck in getJNIEnv

[Sailesh Mukil]

The upstream bug is posted below. I've requested a backport to the HDFS
version Impala depends on, but it hasn't happened yet. I'll make another
push for it.

https://issues.apache.org/jira/browse/HDFS-11851

On Sun, Sep 10, 2017 at 4:01 PM, Jim Apple  wrote:

> It was . bin/set-classpath.sh. Forgot about that one. Thanks, Henry!
>
> On Sun, Sep 10, 2017 at 3:31 PM, Henry Robinson  wrote:
> > I've seen this deadlock before although not in expr-test. I can't
> remember
> > exactly how I cleared it but I believe it was either:
> >
> > 1. make fe && . bin/set-classpath.sh
> > 2. bin/create-test-configuration.sh
> >
> > Sailesh knows about the upstream HDFS bug which I think has been fixed
> but
> > not incorporated into Impala's dependencies.
> >
> > On Sun, Sep 10, 2017 at 1:42 PM Jim Apple  wrote:
> >
> >> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full
> stack
> >> trace:
> >>
> >> #0  __lll_lock_wait () at
> >> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> >> #1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
> >> ) at ../nptl/pthread_mutex_lock.c:80
> >> #2  0x02cf79f6 in mutexLock (m=) at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/os/posix/mutexes.c:28
> >> #3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
> >> stackTrace=0x0) at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/jni_helper.c:581
> >> #4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
> >> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
> >> "loadFileSystems", ap=0x7fff9da0)
> >> at
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/exception.c:183
> >> #5  0x02cf81dd in printExceptionAndFree (env=,
> >> exc=, noPrintFlags=, fmt= >> out>)
> >> at
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/exception.c:213
> >> #6  0x02cf0faf in getGlobalJNIEnv () at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/jni_helper.c:463
> >> #7  getJNIEnv () at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/jni_helper.c:528
> >> #8  0x01a0116a in impala::JniUtil::Init () at
> >> be/src/util/jni-util.cc:105
> >> #9  0x014fd881 in impala::InitCommonRuntime (argc=1,
> >> argv=0x7fffa628, init_jvm=true,
> >> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
> >> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
> >> be/src/exprs/expr-test.cc:7420
> >>
> >> I've tried git fetch, bin/clean.sh, running with the minicluster on,
> >> running with the minicluster off, running with the impala cluster on,
> >> running with it off, running in release mode, debug mode, in gdb, and
> >> out of gdb.
> >>
> >> Has anyone else seen this and escaped from its clutches?
> >>
>

Re: expr-test stuck in getJNIEnv

[Jim Apple]

It was . bin/set-classpath.sh. Forgot about that one. Thanks, Henry!

On Sun, Sep 10, 2017 at 3:31 PM, Henry Robinson  wrote:
> I've seen this deadlock before although not in expr-test. I can't remember
> exactly how I cleared it but I believe it was either:
>
> 1. make fe && . bin/set-classpath.sh
> 2. bin/create-test-configuration.sh
>
> Sailesh knows about the upstream HDFS bug which I think has been fixed but
> not incorporated into Impala's dependencies.
>
> On Sun, Sep 10, 2017 at 1:42 PM Jim Apple  wrote:
>
>> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack
>> trace:
>>
>> #0  __lll_lock_wait () at
>> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> #1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
>> ) at ../nptl/pthread_mutex_lock.c:80
>> #2  0x02cf79f6 in mutexLock (m=) at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
>> #3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
>> stackTrace=0x0) at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
>> #4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
>> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
>> "loadFileSystems", ap=0x7fff9da0)
>> at
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
>> #5  0x02cf81dd in printExceptionAndFree (env=,
>> exc=, noPrintFlags=, fmt=> out>)
>> at
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
>> #6  0x02cf0faf in getGlobalJNIEnv () at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
>> #7  getJNIEnv () at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
>> #8  0x01a0116a in impala::JniUtil::Init () at
>> be/src/util/jni-util.cc:105
>> #9  0x014fd881 in impala::InitCommonRuntime (argc=1,
>> argv=0x7fffa628, init_jvm=true,
>> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
>> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
>> be/src/exprs/expr-test.cc:7420
>>
>> I've tried git fetch, bin/clean.sh, running with the minicluster on,
>> running with the minicluster off, running with the impala cluster on,
>> running with it off, running in release mode, debug mode, in gdb, and
>> out of gdb.
>>
>> Has anyone else seen this and escaped from its clutches?
>>

Re: expr-test stuck in getJNIEnv

[Henry Robinson]

I've seen this deadlock before although not in expr-test. I can't remember
exactly how I cleared it but I believe it was either:

1. make fe && . bin/set-classpath.sh
2. bin/create-test-configuration.sh

Sailesh knows about the upstream HDFS bug which I think has been fixed but
not incorporated into Impala's dependencies.

On Sun, Sep 10, 2017 at 1:42 PM Jim Apple  wrote:

> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack
> trace:
>
> #0  __lll_lock_wait () at
> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
> ) at ../nptl/pthread_mutex_lock.c:80
> #2  0x02cf79f6 in mutexLock (m=) at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
> #3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
> stackTrace=0x0) at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
> #4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
> "loadFileSystems", ap=0x7fff9da0)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
> #5  0x02cf81dd in printExceptionAndFree (env=,
> exc=, noPrintFlags=, fmt= out>)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
> #6  0x02cf0faf in getGlobalJNIEnv () at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
> #7  getJNIEnv () at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
> #8  0x01a0116a in impala::JniUtil::Init () at
> be/src/util/jni-util.cc:105
> #9  0x014fd881 in impala::InitCommonRuntime (argc=1,
> argv=0x7fffa628, init_jvm=true,
> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
> be/src/exprs/expr-test.cc:7420
>
> I've tried git fetch, bin/clean.sh, running with the minicluster on,
> running with the minicluster off, running with the impala cluster on,
> running with it off, running in release mode, debug mode, in gdb, and
> out of gdb.
>
> Has anyone else seen this and escaped from its clutches?
>