I think I included the wrong kind of jstack capture.
Attaching to process ID 24185, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
Deadlock Detection:
No deadlocks found.
Thread 24606: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- jline.internal.NonBlockingInputStream.run() @bci=44, line=278 (Interpreted
frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24603: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.apache.hive.service.cli.session.SessionManager$3.sleepInterval(long)
@bci=1, line=203 (Interpreted frame)
- org.apache.hive.service.cli.session.SessionManager$3.run() @bci=5, line=181
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24600: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
-
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run()
@bci=9, line=3190 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24598: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Interpreted frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Interpreted frame)
- java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403
(Interpreted frame)
- com.jolbox.bonecp.PoolWatchThread.run() @bci=93, line=75 (Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24597: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Interpreted frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=124, line=1093 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=1, line=809 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24596: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
- com.google.common.base.internal.Finalizer.run() @bci=5, line=132
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24593: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Interpreted frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Interpreted frame)
- java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403
(Interpreted frame)
- com.jolbox.bonecp.PoolWatchThread.run() @bci=93, line=75 (Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24592: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Interpreted frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=124, line=1093 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=1, line=809 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24591: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
- com.google.common.base.internal.Finalizer.run() @bci=5, line=132
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24590: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.apache.derby.impl.services.daemon.BasicDaemon.rest() @bci=22
(Interpreted frame)
- org.apache.derby.impl.services.daemon.BasicDaemon.run() @bci=22 (Interpreted
frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
Thread 24589: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=28, line=526 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)
Thread 24553: (state = BLOCKED)
Thread 24552: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted
frame)
Thread 24551: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
- java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191
(Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Interpreted
frame)
Thread 24521: (state = IN_NATIVE)
- java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame;
information may be imprecise)
- java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled
frame)
- java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)
- java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)
- jline.internal.NonBlockingInputStream.read(long, boolean) @bci=102, line=169
(Interpreted frame)
- jline.internal.NonBlockingInputStream.read(long) @bci=27, line=137
(Interpreted frame)
- jline.internal.NonBlockingInputStream.read(byte[], int, int) @bci=51,
line=246 (Interpreted frame)
- jline.internal.InputStreamReader.read(char[], int, int) @bci=177, line=261
(Interpreted frame)
- jline.internal.InputStreamReader.read() @bci=32, line=198 (Interpreted frame)
- jline.console.ConsoleReader.readCharacter() @bci=4, line=2145 (Interpreted
frame)
- jline.console.ConsoleReader.readLine(java.lang.String, java.lang.Character)
@bci=202, line=2349 (Interpreted frame)
- jline.console.ConsoleReader.readLine(java.lang.String) @bci=3, line=2269
(Interpreted frame)
- org.apache.hive.beeline.BeeLine.execute(jline.console.ConsoleReader,
boolean) @bci=34, line=833 (Interpreted frame)
- org.apache.hive.beeline.BeeLine.begin(java.lang.String[],
java.io.InputStream) @bci=85, line=793 (Interpreted frame)
- org.apache.hive.beeline.BeeLine.mainWithInputRedirection(java.lang.String[],
java.io.InputStream) @bci=11, line=493 (Interpreted frame)
- org.apache.hive.beeline.BeeLine.main(java.lang.String[]) @bci=2, line=476
(Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method,
java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=100, line=62 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
@bci=56, line=498 (Interpreted frame)
- org.apache.hadoop.util.RunJar.run(java.lang.String[]) @bci=450, line=233
(Interpreted frame)
- org.apache.hadoop.util.RunJar.main(java.lang.String[]) @bci=8, line=148
(Interpreted frame)
________________________________
From: Shawn Weeks <swe...@weeksconsulting.us>
Sent: Tuesday, February 19, 2019 6:19 PM
To: user@hive.apache.org
Subject: Beeline Hive Embedded Locks Up on Add Jar
Hi, I'm trying to setup an embedded Hive environment to test code deployments.
I've got custom hive-site.xml and related files such that when I start Beeline
with beeline -u "jdbc:hive2:///" I'm connecting to Derby Metastore in the local
directory and reading and writing from the local filesystem instead of HDFS. I
can create schemas, tables, insert records, query records etc all locally
without HDFS or Yarn. However if I try and run add jar for a custom class I'm
working on Beeline acknowledges that I added the jar but never returns a prompt
forcing me to ctrl+c out of beeline. I'm guessing it's causing some sort of
deadlock with the embedded instance of Hive but I'm not sure how to fix it.
This is Hive 1.2.1 version included in HDP 2.6.5 on RedHat 7. The same commands
run just fine in the old Hive CLI against the same metastore. I'm attaching a
jstack trace as well. Not sure if this is a bug or a limitation.
Thanks
Shawn Weeks
Here is the debug logs from Beeline
0: jdbc:hive2:///> add jar
file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar;
19/02/19 18:09:26 [main]: INFO session.HiveSessionImpl: We are setting the
hadoop caller context to f49cfcc3-3653-4ac3-b5be-364407e11589 for thread main
19/02/19 18:09:26 [main]: INFO operation.Operation: Putting temp output to file
/tmp/1454256952/f49cfcc3-3653-4ac3-b5be-364407e115896013231645321376050.pipeout
19/02/19 18:09:26 [main]: DEBUG parse.VariableSubstitution: Substitution is on:
jar file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar
Added [file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar] to
class path
19/02/19 18:09:26 [main]: INFO SessionState: Added
[file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar] to class path
Added resources:
[file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar]
19/02/19 18:09:26 [main]: INFO SessionState: Added resources:
[file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar]
19/02/19 18:09:26 [main]: INFO session.HiveSessionImpl: We are resetting the
hadoop caller context for thread main
19/02/19 18:09:26 [main]: DEBUG cli.CLIService: SessionHandle
[f49cfcc3-3653-4ac3-b5be-364407e11589]: executeStatementAsync()
19/02/19 18:09:26 [main]: DEBUG jdbc.HiveStatement: requesting progress
response :true
19/02/19 18:09:26 [main]: DEBUG cli.CLIService: OperationHandle
[opType=EXECUTE_STATEMENT,
getHandleIdentifier()=0a2843cf-5295-4bca-add6-86148099c290]:
getOperationStatus()
19/02/19 18:09:26 [main]: DEBUG cli.CLIService: Sending Progress Update
Response :JobProgressUpdate{progressedPercentage=0.0, footerSummary='',
startTimeMillis=0, headers='', rows='', status=''}
19/02/19 18:09:26 [main]: DEBUG logs.InPlaceUpdateStream$EventNotifier:
progress bar is complete
19/02/19 18:09:26 [main]: DEBUG jdbc.HiveStatement: Progress Bar Response
Received: TProgressUpdateResp(headerNames:[], rows:[],
progressedPercentage:0.0, status:NOT_AVAILABLE, footerSummary:, startTime:0)
19/02/19 18:09:26 [main]: DEBUG logs.InPlaceUpdateStream$EventNotifier:
progress bar is complete
19/02/19 18:09:26 [Thread-36]: INFO session.HiveSessionImpl: We are setting the
hadoop caller context to f49cfcc3-3653-4ac3-b5be-364407e11589 for thread
Thread-36
19/02/19 18:09:26 [Thread-36]: INFO session.HiveSessionImpl: We are resetting
the hadoop caller context for thread Thread-36
19/02/19 18:09:26 [Thread-36]: DEBUG cli.CLIService: OperationHandle
[opType=EXECUTE_STATEMENT,
getHandleIdentifier()=0a2843cf-5295-4bca-add6-86148099c290]: fetchResults()
INFO : Added
[file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar] to class path
INFO : Added resources:
[file:///home/1454256952/Projects/hive/lib/hive-custom-io-1.0.jar]
19/02/19 18:09:26 [Thread-36]: INFO session.HiveSessionImpl: We are setting the
hadoop caller context to f49cfcc3-3653-4ac3-b5be-364407e11589 for thread
Thread-36
19/02/19 18:09:26 [Thread-36]: INFO session.HiveSessionImpl: We are resetting
the hadoop caller context for thread Thread-36
19/02/19 18:09:26 [Thread-36]: DEBUG cli.CLIService: OperationHandle
[opType=EXECUTE_STATEMENT,
getHandleIdentifier()=0a2843cf-5295-4bca-add6-86148099c290]: fetchResults()
19/02/19 18:09:26 [main]: DEBUG jdbc.HiveStatement: requesting progress
response :true
19/02/19 18:09:26 [main]: DEBUG logs.InPlaceUpdateStream$EventNotifier:
progress bar is complete
No rows affected (0.013 seconds)
19/02/19 18:09:26 [main]: INFO session.HiveSessionImpl: We are setting the
hadoop caller context to f49cfcc3-3653-4ac3-b5be-364407e11589 for thread main
19/02/19 18:09:26 [main]: INFO session.HiveSessionImpl: We are resetting the
hadoop caller context for thread main
19/02/19 18:09:26 [main]: DEBUG cli.CLIService: OperationHandle
[opType=EXECUTE_STATEMENT,
getHandleIdentifier()=0a2843cf-5295-4bca-add6-86148099c290]: fetchResults()
19/02/19 18:09:26 [main]: INFO session.HiveSessionImpl: We are setting the
hadoop caller context to f49cfcc3-3653-4ac3-b5be-364407e11589 for thread main
-- I hit ctrl+c here after waiting for several minutes --
-bash-4.2$
