GitHub user mahalingamagesthian edited a discussion: Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working
Hi @amanraj2520 , @pan3793 I wish that you are doing really well. We have a scenario to run spark-submit with kyuubi-spark-authz-shaded jar. We have figured out that the Sparkdriver kept running after the completion of the spark programme execution. We are seeing nondaemon threads similar to #6212 & #6233 . Inspired by this, we are very quick to download kyuubi-spark-authz-shaded-2.12.1.10.1.jar. We don't find it fixing the issue. The sparkdriver logs doesn't have a mention of the Logs in line https://github.com/amanraj2520/kyuubi/blob/auditShutdown/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/SparkRangerAdminPlugin.scala#L79 I doubt if the fix is working or Am I missing anything ? I am happy to provide additional information from the test scenario. Spark-submit is a batch on K8s. Spark submit connects to Apache Ranger download the policy evaluate the policy. kyuubi-spark-authz-shaded.jar is a key module to perform this operation. JSTACK thread: ``` "zkConnectionManagerCallback-5-thread-1" #113 prio=5 os_prio=0 cpu=0.90ms elapsed=26.68s tid=0x00007f1de0004000 nid=0xbd waiting on condition [0x00007f1dfa1d6000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000007111bdff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take([email protected]/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at java.lang.Thread.run([email protected]/Thread.java:829) ``` GitHub link: https://github.com/apache/kyuubi/discussions/6996 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
