[
https://issues.apache.org/jira/browse/TRAFODION-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Subbiah updated TRAFODION-1107:
--------------------------------------
Fix Version/s: (was: 2.2.0)
> LP Bug: 1438466 - Multiple tdm_arkcmp child processes started after receipt
> of HBase error
> ------------------------------------------------------------------------------------------
>
> Key: TRAFODION-1107
> URL: https://issues.apache.org/jira/browse/TRAFODION-1107
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
> Reporter: Joanie Cooper
> Assignee: Qifan Chen
> Priority: Critical
>
> During a fresh test of running the compGeneral regression suite
> while artificially interjecting an error return from the TrxRegionEndpoint
> coprocessor, numerous tdm_arkcmp child processes were started.
> Before the error hit, we seemed to have a normal number of compilers
> [$Z0005MG] 000,00003170 001 GEN ES--A-- $Z0002KK $Z0002IV tdm_arkcmp
>
> [$Z0005MG] 000,00003292 001 GEN ES--A-- $Z0002P2 $Z0002KK tdm_arkcmp
>
> [$Z0005MG] 000,00003816 001 GEN ES--A-- $Z000341 $Z0002P2 tdm_arkcmp
>
> [$Z0005MG] 000,00003886 001 GEN ES--A-- $Z000361 $Z000341 tdm_arkcmp
> After forcing the error, it looks like we have new compilers being generated,
> all ultimately part of the original tdm_arkcmp parent off of the sqlci
> session.
> This is a result of a drop statement. From the sqlci window, the
> statement appears hung, as it never returns. But, it appears the
> compilers keep generating new children and the query ultimately never returns.
> When I killed the query, it had 174 compilers running.
> I tried a pstack for one of the compilers, I’ve attached it below.
> g4t3037{joaniec}3: sqps
> Processing cluster.conf on local host g4t3037.houston.hp.com
> [$Z000AF9] Shell/shell Version 1.0.1 Release 1.1.0 (Build release [joaniec],
> date 26Mar15)
> [$Z000AF9] %ps
> [$Z000AF9] NID,PID(os) PRI TYPE STATES NAME PARENT PROGRAM
> [$Z000AF9] ------------ --- ---- ------- ----------- -----------
> ---------------
> [$Z000AF9] 000,00018562 000 WDG ES--A-- $WDG000 NONE sqwatchdog
>
> [$Z000AF9] 000,00018563 000 PSD ES--A-- $PSD000 NONE pstartd
>
> [$Z000AF9] 000,00018592 001 DTM ES--A-- $TM0 NONE tm
>
> [$Z000AF9] 000,00019243 001 GEN ES--A-- $ZSC000 NONE mxsscp
>
> [$Z000AF9] 000,00019274 001 SSMP ES--A-- $ZSM000 NONE mxssmp
>
> [$Z000AF9] 000,00020982 001 GEN ES--A-- $ZLOBSRV0 NONE mxlobsrvr
>
> [$Z000AF9] 000,00007356 001 GEN ES--A-- $Z000606 NONE sqlci
>
> [$Z000AF9] 000,00007416 001 GEN ES--A-- $Z00061W $Z000606 tdm_arkcmp
>
> [$Z000AF9] 000,00007960 001 GEN ES--A-- $Z0006HF $Z00061W tdm_arkcmp
>
> [$Z000AF9] 000,00008021 001 GEN ES--A-- $Z0006J6 $Z0006HF tdm_arkcmp
>
> [$Z000AF9] 000,00008079 001 GEN ES--A-- $Z0006KU $Z0006J6 tdm_arkcmp
>
> [$Z000AF9] 000,00008137 001 GEN ES--A-- $Z0006MH $Z0006KU tdm_arkcmp
>
> [$Z000AF9] 000,00008194 001 GEN ES--A-- $Z0006P4 $Z0006MH tdm_arkcmp
>
> [$Z000AF9] 000,00008252 001 GEN ES--A-- $Z0006QS $Z0006P4 tdm_arkcmp
>
> [$Z000AF9] 000,00008312 001 GEN ES--A-- $Z0006SH $Z0006QS tdm_arkcmp
>
> [$Z000AF9] 000,00008369 001 GEN ES--A-- $Z0006U4 $Z0006SH tdm_arkcmp
>
> [$Z000AF9] 000,00008427 001 GEN ES--A-- $Z0006VS $Z0006U4 tdm_arkcmp
>
> [$Z000AF9] 000,00008491 001 GEN ES--A-- $Z0006XL $Z0006VS tdm_arkcmp
>
> [$Z000AF9] 000,00009023 001 GEN ES--A-- $Z0007CT $Z0006XL tdm_arkcmp
>
> [$Z000AF9] 000,00009081 001 GEN ES--A-- $Z0007EG $Z0007CT tdm_arkcmp
>
> [$Z000AF9] 000,00009141 001 GEN ES--A-- $Z0007G6 $Z0007EG tdm_arkcmp
>
> [$Z000AF9] 000,00009202 001 GEN ES--A-- $Z0007HX $Z0007G6 tdm_arkcmp
>
> [$Z000AF9] 000,00009262 001 GEN ES--A-- $Z0007JM $Z0007HX tdm_arkcmp
>
> [$Z000AF9] 000,00009320 001 GEN ES--A-- $Z0007LA $Z0007JM tdm_arkcmp
>
> [$Z000AF9] 000,00009489 001 GEN ES--A-- $Z0007R4 $Z0007LA tdm_arkcmp
>
> [$Z000AF9] 000,00009547 001 GEN ES--A-- $Z0007SS $Z0007R4 tdm_arkcmp
>
> [$Z000AF9] 000,00009604 001 GEN ES--A-- $Z0007UE $Z0007SS tdm_arkcmp
>
> [$Z000AF9] 000,00009661 001 GEN ES--A-- $Z0007W1 $Z0007UE tdm_arkcmp
>
> [$Z000AF9] 000,00009728 001 GEN ES--A-- $Z0007XY $Z0007W1 tdm_arkcmp
>
> [$Z000AF9] 000,00010268 001 GEN ES--A-- $Z0008DD $Z0007XY tdm_arkcmp
>
> [$Z000AF9] 000,00010364 001 GEN ES--A-- $Z0008G4 $Z0008DD tdm_arkcmp
>
> [$Z000AF9] 000,00010421 001 GEN ES--A-- $Z0008HR $Z0008G4 tdm_arkcmp
>
> [$Z000AF9] 000,00010479 001 GEN ES--A-- $Z0008JE $Z0008HR tdm_arkcmp
>
> [$Z000AF9] 000,00010537 001 GEN ES--A-- $Z0008L2 $Z0008JE tdm_arkcmp
>
> [$Z000AF9] 000,00010620 001 GEN ES--A-- $Z0008NF $Z0008L2 tdm_arkcmp
>
> [$Z000AF9] 000,00010677 001 GEN ES--A-- $Z0008Q2 $Z0008NF tdm_arkcmp
>
> [$Z000AF9] 000,00010735 001 GEN ES--A-- $Z0008RQ $Z0008Q2 tdm_arkcmp
>
> [$Z000AF9] 000,00010796 001 GEN ES--A-- $Z0008TG $Z0008RQ tdm_arkcmp
>
> [$Z000AF9] 000,00010857 001 GEN ES--A-- $Z0008V7 $Z0008TG tdm_arkcmp
>
> [$Z000AF9] 000,00011367 001 GEN ES--A-- $Z00099S $Z0008V7 tdm_arkcmp
>
> [$Z000AF9] 000,00011437 001 GEN ES--A-- $Z0009BS $Z00099S tdm_arkcmp
>
> [$Z000AF9] 000,00011496 001 GEN ES--A-- $Z0009DG $Z0009BS tdm_arkcmp
>
> [$Z000AF9] 000,00011558 001 GEN ES--A-- $Z0009F8 $Z0009DG tdm_arkcmp
>
> [$Z000AF9] 000,00011618 001 GEN ES--A-- $Z0009GY $Z0009F8 tdm_arkcmp
>
> [$Z000AF9] 000,00011680 001 GEN ES--A-- $Z0009IQ $Z0009GY tdm_arkcmp
>
> [$Z000AF9] 000,00011741 001 GEN ES--A-- $Z0009KG $Z0009IQ tdm_arkcmp
>
> [$Z000AF9] 000,00011801 001 GEN ES--A-- $Z0009M6 $Z0009KG tdm_arkcmp
>
> [$Z000AF9] 000,00011862 001 GEN ES--A-- $Z0009NX $Z0009M6 tdm_arkcmp
>
> [$Z000AF9] 000,00011921 001 GEN ES--A-- $Z0009QL $Z0009NX tdm_arkcmp
>
> [$Z000AF9] 000,00012410 001 GEN ES--A-- $Z000A4K $Z0009QL tdm_arkcmp
>
> [$Z000AF9] 000,00012509 001 GEN ES--A-- $Z000A7E $Z000A4K tdm_arkcmp
>
> [$Z000AF9] 000,00012582 001 GEN ES--A-- $Z000A9H $Z000A7E tdm_arkcmp
>
> [$Z000AF9] 000,00012646 001 GEN ES--A-- $Z000ABB $Z000A9H tdm_arkcmp
>
> [$Z000AF9] 000,00012703 001 GEN ES--A-- $Z000ACY $Z000ABB tdm_arkcmp
>
> [$Z000AF9] 000,00012760 001 GEN ES--A-- $Z000AEK $Z000ACY tdm_arkcmp
>
> [$Z000AF9] 000,00012784 001 GEN ES--A-- $Z000AF9 NONE shell
>
> [$Z000AF9] 001,00018560 000 WDG ES--A-- $WDG001 NONE sqwatchdog
>
> [$Z000AF9] 001,00018561 000 PSD ES--A-- $PSD001 NONE pstartd
>
> [$Z000AF9] 001,00018640 001 DTM ES--A-- $TM1 NONE tm
>
> [$Z000AF9] 001,00019258 001 GEN ES--A-- $ZSC001 NONE mxsscp
>
> [$Z000AF9] 001,00019289 001 SSMP ES--A-- $ZSM001 NONE mxssmp
>
> [$Z000AF9] 001,00020989 001 GEN ES--A-- $ZLOBSRV1 NONE mxlobsrvr
> [$Z000GHJ] 000,00020170 001 GEN ES--A-- $Z000GGA $Z000GEK tdm_arkcmp
>
> [$Z000GHJ] 000,00020214 001 GEN ES--A-- $Z000GHJ NONE shell
>
> [$Z000GHJ] 001,00018560 000 WDG ES--A-- $WDG001 NONE sqwatchdog
>
> [$Z000GHJ] 001,00018561 000 PSD ES--A-- $PSD001 NONE pstartd
>
> [$Z000GHJ] 001,00018640 001 DTM ES--A-- $TM1 NONE tm
>
> [$Z000GHJ] 001,00019258 001 GEN ES--A-- $ZSC001 NONE mxsscp
>
> [$Z000GHJ] 001,00019289 001 SSMP ES--A-- $ZSM001 NONE mxssmp
>
> [$Z000GHJ] 001,00020989 001 GEN ES--A-- $ZLOBSRV1 NONE mxlobsrvr
>
> g4t3037{joaniec}120: pstack $Z000GGA
> Usage: pstack <process-id>
> g4t3037{joaniec}121: pstack 20110
> Thread 30 (Thread 0x7fffee0a7700 (LWP 20111)):
> #0 0x0000003e3c00e84d in accept () from /lib64/libpthread.so.0
> #1 0x00007fffefec1995 in SB_Trans::Sock_Listener::accept() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #2 0x00007fffefec84e6 in SB_Trans::Sock_Stream_Accept_Thread::run() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #3 0x00007fffefec8213 in sock_stream_accept_thread_fun(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #5 0x00007fffee540fa7 in thread_fun(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #6 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 29 (Thread 0x7fffed6a6700 (LWP 20112)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007fffee5435e2 in SB_Thread::CV::wait() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #2 0x00007fffee5436be in SB_Thread::CV::wait(bool) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #3 0x00007fffefebac47 in SB_Sig_Queue::remove() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4 0x00007fffefec8977 in SB_Trans::Sock_Stream_Helper_Thread::run() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #5 0x00007fffefec823a in sock_helper_thread_fun(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #6 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #7 0x00007fffee540fa7 in thread_fun(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #8 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 28 (Thread 0x7fffead82700 (LWP 20113)):
> #0 0x0000003e3bc3399d in sigtimedwait () from /lib64/libc.so.6
> #1 0x00007fffefe865c4 in local_monitor_reader(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #2 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #3 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 27 (Thread 0x7fffea301700 (LWP 20115)):
> #0 0x0000003e3c00ed2d in nanosleep () from /lib64/libpthread.so.0
> #1 0x00007fffef6032b9 in Sleep(unsigned int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libwin.so
> #2 0x00007ffff7747d27 in memMonitorUpdateThread(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #3 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #4 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 26 (Thread 0x7fffe977e700 (LWP 20116)):
> #0 0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007fffefec0cd5 in SB_Trans::Sock_Controller::epoll_wait(char const*,
> int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #2 0x00007fffefec01f3 in SB_Trans::Sock_Comp_Thread::run() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #3 0x00007fffefec002d in sock_comp_thread_fun(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4 0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #5 0x00007fffee540fa7 in thread_fun(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #6 0x00007fffee54421c in sb_thread_sthr_disp(void*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 25 (Thread 0x7fffe523e700 (LWP 20117)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 24 (Thread 0x7fffe513d700 (LWP 20118)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 23 (Thread 0x7fffe503c700 (LWP 20119)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 22 (Thread 0x7fffe4f3b700 (LWP 20120)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 21 (Thread 0x7fffe4e3a700 (LWP 20121)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 20 (Thread 0x7fffe4d39700 (LWP 20122)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 19 (Thread 0x7fffe4c38700 (LWP 20123)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 18 (Thread 0x7fffe4b37700 (LWP 20124)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 17 (Thread 0x7fffe4a36700 (LWP 20125)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 16 (Thread 0x7fffe4935700 (LWP 20126)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff15210a8 in GCTaskThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 15 (Thread 0x7fffdfdec700 (LWP 20127)):
> #0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e88ef in os::PlatformEvent::park(long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff1970169 in VMThread::loop() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff1970470 in VMThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 14 (Thread 0x7fffdfceb700 (LWP 20128)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff1647378 in JVM_MonitorWait () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007fffe56ba7f8 in ?? ()
> #5 0x00007fffdfcd3390 in ?? ()
> #6 0x0000000000c33ca0 in ?? ()
> #7 0x00007fffdfcd31c0 in ?? ()
> #8 0x00007fffdfcd3158 in ?? ()
> #9 0x0000000000000000 in ?? ()
> Thread 13 (Thread 0x7fffdfbea700 (LWP 20129)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff1647378 in JVM_MonitorWait () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007fffe56ba7f8 in ?? ()
> #5 0x00007fffdfbd22e0 in ?? ()
> #6 0x0000000000c3e000 in ?? ()
> #7 0x0000000000c38210 in ?? ()
> #8 0x00007fffdfbd2268 in ?? ()
> #9 0x0000000000000000 in ?? ()
> Thread 12 (Thread 0x7fffdfae9700 (LWP 20130)):
> #0 0x0000003e3c00d720 in sem_wait () from /lib64/libpthread.so.0
> #1 0x00007ffff17e85ca in check_pending_signals(bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17e1835 in signal_thread_entry(JavaThread*, Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff1922ea5 in JavaThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #7 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 11 (Thread 0x7fffdf9e8700 (LWP 20131)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff1414ad8 in CompileQueue::get() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x00007ffff1922ea5 in JavaThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #8 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 10 (Thread 0x7fffdf8e7700 (LWP 20132)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff1414ad8 in CompileQueue::get() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x00007ffff1922ea5 in JavaThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #8 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #9 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 9 (Thread 0x7fffdf7e6700 (LWP 20133)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e3243 in os::PlatformEvent::park() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff1879388 in ServiceThread::service_thread_entry(JavaThread*,
> Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff1922d9f in JavaThread::thread_main_inner() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff1922ea5 in JavaThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #8 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #9 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 8 (Thread 0x7fffdf6e5700 (LWP 20134)):
> #0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e88ef in os::PlatformEvent::park(long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff191edc1 in WatcherThread::sleep() const () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff191f2de in WatcherThread::run() () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff17e9988 in java_start(Thread*) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7 0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 7 (Thread 0x7fffde317700 (LWP 20137)):
> #0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e8114 in Parker::park(bool, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff1946a05 in Unsafe_Park () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007fffe56ba7f8 in ?? ()
> #4 0x0000000000bb4968 in ?? ()
> #5 0x0000000000000000 in ?? ()
> Thread 6 (Thread 0x7fffddedb700 (LWP 20141)):
> #0 0x0000003e3bcdf253 in poll () from /lib64/libc.so.6
> #1 0x00007fffde32a2ca in
> Java_org_apache_hadoop_net_unix_DomainSocketWatcher_doPoll0 () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/sql/local_hadoop/hadoop-2.5.0-cdh5.3.0/lib/native/libhadoop.so.1.0.0
> #2 0x00007fffe56ba7f8 in ?? ()
> #3 0x00000000020a4800 in ?? ()
> #4 0x0000000000000000 in ?? ()
> Thread 5 (Thread 0x7fffde2de700 (LWP 20153)):
> #0 0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007fffde96b707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from
> /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/libnio.so
> #2 0x00007fffe56ba7f8 in ?? ()
> #3 0x0000000000000000 in ?? ()
> Thread 4 (Thread 0x7fffddbd8700 (LWP 20154)):
> #0 0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e8228 in Parker::park(bool, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff1946a05 in Unsafe_Park () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007fffe56ba7f8 in ?? ()
> #4 0x00007fffe56ae058 in ?? ()
> #5 0x0000000000000000 in ?? ()
> Thread 3 (Thread 0x7fffddfdc700 (LWP 20156)):
> #0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e8114 in Parker::park(bool, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff1946a05 in Unsafe_Park () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007fffe56ba7f8 in ?? ()
> #4 0x00007fffe56ae058 in ?? ()
> #5 0x0000000000000000 in ?? ()
> Thread 2 (Thread 0x7fffdddda700 (LWP 20236)):
> #0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007ffff17e8114 in Parker::park(bool, long) () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff1946a05 in Unsafe_Park () from
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3 0x00007fffe56ba7f8 in ?? ()
> #4 0x00007fffe56ae058 in ?? ()
> #5 0x0000000000000000 in ?? ()
> Thread 1 (Thread 0x7fffee315c00 (LWP 20110)):
> #0 0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007fffee543778 in SB_Thread::CV::wait(int, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #2 0x00007fffee54383d in SB_Thread::CV::wait(bool, int, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #3 0x00007fffefe988cb in SB_Ms_Event_Mgr::wait(long) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4 0x00007fffefeb916e in XWAIT_com(short, int, bool) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #5 0x00007fffefeb8f1f in XWAIT(short, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #6 0x00007fffefc3731a in fs_int_fs_file_awaitiox(short*, void**, int*,
> long*, int, short*, bool, bool) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
> #7 0x00007fffefc30e15 in BAWAITIOXTS(short*, void**, int*, long*, int,
> short*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
> #8 0x00007ffff70be1f7 in GuaConnectionToServer::wait(int, unsigned int*,
> IpcAwaitiox*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #9 0x00007ffff70b51cb in IpcSetOfConnections::waitOnSet(int, int, int*) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #10 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #11 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #12 0x00007ffff70bdf42 in GuaConnectionToServer::tryToStartNewIO() () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #13 0x00007ffff70beb38 in GuaConnectionToServer::send(IpcMessageBuffer*) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #14 0x00007ffff70b72e4 in IpcMessageStream::send(int, long) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #15 0x00007ffff773fad5 in ExSqlComp::sendRequest(CmpMessageObj*, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #16 0x00007ffff773e620 in
> ExSqlComp::sendRequest(CmpMessageObj::MessageTypeEnum, char const*, unsigned
> int, int, long*, int, int, char const*, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #17 0x00007ffff775fdd9 in CliStatement::prepare2(char*, ComDiagsArea&, char*,
> unsigned int, int, int, unsigned int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #18 0x00007ffff7760846 in CliStatement::prepare(char*, ComDiagsArea&, char*,
> unsigned int, int, int, unsigned int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #19 0x00007ffff77130d4 in SQLCLI_Prepare2 () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #20 0x00007ffff776dc6e in SQL_EXEC_Prepare2 () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #21 0x00007ffff3c819d2 in ExeCliInterface::prepare(char const*,
> SQLMODULE_ID_t*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*,
> SQLCLI_OBJ_ID*, char**, Queue*, char**, Queue*, char*, int*,
> SQL_QUERY_COST_INFO*, SQL_QUERY_COMPILER_STATS_INFO*, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
> #22 0x00007ffff3c83460 in ExeCliInterface::fetchRowsPrologue(char const*,
> int, int, char*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
> #23 0x00007ffff7a627fb in
> CmpSeabaseDDL::getObjectUIDandOwners(ExeCliInterface*, char const*, char
> const*, char const*, ComObjectType, int&, int&, bool, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #24 0x00007ffff7ae763d in CmpSeabaseDDL::getSeabaseUserTableDesc(NAString
> const&, NAString const&, NAString const&, ComObjectType, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #25 0x00007ffff7aea687 in CmpSeabaseDDL::getSeabaseTableDesc(NAString const&,
> NAString const&, NAString const&, ComObjectType, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #26 0x00007ffff4d5d304 in NATableDB::get(CorrName&, BindWA*, desc_struct*) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #27 0x00007ffff4ab2efc in BindWA::getNATable(CorrName&, int, desc_struct*) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #28 0x00007ffff4ad361e in Scan::bindNode(BindWA*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #29 0x00007ffff4a9fe07 in RelExpr::bindChildren(BindWA*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #30 0x00007ffff4adffde in RelRoot::bindNode(BindWA*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #31 0x00007ffff7a3f50e in CmpMain::compile(char const*, int, RelExpr*&,
> char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int,
> int, int*, timeval*, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #32 0x00007ffff7a4232c in CmpMain::sqlcomp(char const*, int, RelExpr*&,
> char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int,
> int, int*, timeval*, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #33 0x00007ffff7a43630 in CmpMain::sqlcomp(QueryText&, int, char**, unsigned
> int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #34 0x00007ffff40dfc08 in CmpStatement::process(CmpMessageSQLText const&) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
> #35 0x00007ffff40ca93d in ExCmpMessage::actOnReceive(IpcConnection*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
> #36 0x00007ffff70a58d6 in
> IpcMessageStream::internalActOnReceive(IpcMessageBuffer*, IpcConnection*) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #37 0x00007ffff70c02fb in
> GuaConnectionToClient::acceptBuffer(IpcMessageBuffer*, unsigned int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #38 0x00007ffff70c100f in GuaReceiveControlConnection::wait(int, unsigned
> int*, IpcAwaitiox*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #39 0x00007ffff70c23b2 in GuaConnectionToClient::wait(int, unsigned int*,
> IpcAwaitiox*) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #40 0x00007ffff70b54aa in IpcSetOfConnections::waitOnSet(int, int, int*) ()
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #41 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #42 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #43 0x00000000004064ae in main ()
> One more quick note, even after the sqlci session ended,
> only the initial tdm_arkcmp process was cleaned up.
> The others remained:
> Processing cluster.conf on local host g4t3037.houston.hp.com
> [$Z000IRF] Shell/shell Version 1.0.1 Release 1.1.0 (Build release [joaniec],
> dat
> e 26Mar15)
> [$Z000IRF] %ps
> [$Z000IRF] NID,PID(os) PRI TYPE STATES NAME PARENT PROGRAM
> [$Z000IRF] ------------ --- ---- ------- ----------- -----------
> ---------------
> [$Z000IRF] 000,00018562 000 WDG ES--A-- $WDG000 NONE sqwatchdog
>
> [$Z000IRF] 000,00018563 000 PSD ES--A-- $PSD000 NONE pstartd
>
> [$Z000IRF] 000,00018592 001 DTM ES--A-- $TM0 NONE tm
>
> [$Z000IRF] 000,00019243 001 GEN ES--A-- $ZSC000 NONE mxsscp
>
> [$Z000IRF] 000,00019274 001 SSMP ES--A-- $ZSM000 NONE mxssmp
>
> [$Z000IRF] 000,00020982 001 GEN ES--A-- $ZLOBSRV0 NONE mxlobsrvr
>
> [$Z000IRF] 000,00007416 001 GEN ES--A-- $Z00061W NONE tdm_arkcmp
>
> [$Z000IRF] 000,00007960 001 GEN ES--A-- $Z0006HF $Z00061W tdm_arkcmp
>
> [$Z000IRF] 000,00008021 001 GEN ES--A-- $Z0006J6 $Z0006HF tdm_arkcmp
>
> [$Z000IRF] 000,00008079 001 GEN ES--A-- $Z0006KU $Z0006J6 tdm_arkcmp
>
> [$Z000IRF] 000,00008137 001 GEN ES--A-- $Z0006MH $Z0006KU tdm_arkcmp
>
> [$Z000IRF] 000,00008194 001 GEN ES--A-- $Z0006P4 $Z0006MH tdm_arkcmp
>
> [$Z000IRF] 000,00008252 001 GEN ES--A-- $Z0006QS $Z0006P4 tdm_arkcmp
>
> [$Z000IRF] 000,00008312 001 GEN ES--A-- $Z0006SH $Z0006QS tdm_arkcmp
>
> [$Z000IRF] 000,00008369 001 GEN ES--A-- $Z0006U4 $Z0006SH tdm_arkcmp
>
> [$Z000IRF] 000,00008427 001 GEN ES--A-- $Z0006VS $Z0006U4 tdm_arkcmp
>
> …
> The new memoryUsage threshold and throttle capabilities were used to initially
> see the problem.
> This was on the CDH 5.3 workstation using the install_local_hadoop support
> on Trafidion R1.1.0 build version git032415 or later build.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
