Hi Spark Team Some of my spark applications on K8s ended with the below error . These applications though completed successfully (as per the event log SparkListenerApplicationEnd event at the end) stil have even files with .inprogress. This causes the application to be shown as inprogress in SHS.
Spark v : 3.0.1 22/03/31 08:33:34 WARN ShutdownHookManager: ShutdownHook '$anon$2' timeout, java.util.concurrent.TimeoutException java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:68) 22/03/31 08:33:34 WARN SparkContext: Ignoring Exception while stopping SparkContext from shutdown hook java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475) at org.apache.spark.util.ThreadUtils$.shutdown(ThreadUtils.scala:348) Please let me know if there is a solution for it .. Regards Pralabh Kumar