[jira] [Updated] (SPARK-26587) Deadlock between SparkUI thread and Driver thread
[
https://issues.apache.org/jira/browse/SPARK-26587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-26587:
-
Attachment: _Spark_node_hanging__Thread_dump_from_application_master.txt
> Deadlock between SparkUI thread and Driver thread
> ---
>
> Key: SPARK-26587
> URL: https://issues.apache.org/jira/browse/SPARK-26587
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.2.0
> Environment: EMR 5.9.0
>Reporter: Vitaliy Savkin
>Priority: Major
> Attachments:
> _Spark_node_hanging__Thread_dump_from_application_master.txt
>
>
> One time in a month (~1000 runs) one of our spark applications freezes at
> startup. jstack says that there is a deadlock. Please see locks
> 0x802c00c0 and 0x8271bb98 in stacktraces below.
> {noformat}
> "Driver":
> at java.lang.Package.getSystemPackage(Package.java:540)
> - waiting to lock <0x802c00c0> (a java.util.HashMap)
> at java.lang.ClassLoader.getPackage(ClassLoader.java:1625)
> at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:394)
> at java.net.URLClassLoader.definePackageInternal(URLClassLoader.java:420)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:452)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> - locked <0x82789598> (a
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:221)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:210)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> - locked <0x82789540> (a
> org.apache.spark.sql.internal.NonClosableMutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:294)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:289)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2516)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405)
> - locked <0x8271bb98> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:981)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2189)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2702)
> at
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:599)
> at java.net.URL.(URL.java:490)
> at java.net.URL.(URL.java:439)
> at java.net.JarURLConnection.parseSpecs(JarURLConnection.java:175)
> at java.net.JarURLConnection.(JarURLConnection.java:158)
> at sun.net.www.protocol.jar.JarURLConnection.(JarURLConnection.java:81)
> at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:41)
> at java.net.URL.openConnection(URL.java:979)
> at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:216)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:210)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> - locked <0x82789540> (a
> org.apache.spark.sql.internal.NonClosableMutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:262)
> at
> org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:362)
> at
> org.apache.spark.sql.hive.HiveUtils$.newClientForMetada
[jira] [Commented] (SPARK-26587) Deadlock between SparkUI thread and Driver thread
[
https://issues.apache.org/jira/browse/SPARK-26587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752908#comment-16752908
]
Umayr Hassan commented on SPARK-26587:
--
[~Vitaliy.Savkin] we see something similar in our apps. Any breakthrough?
[^_Spark_node_hanging__Thread_dump_from_application_master.txt]
> Deadlock between SparkUI thread and Driver thread
> ---
>
> Key: SPARK-26587
> URL: https://issues.apache.org/jira/browse/SPARK-26587
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.2.0
> Environment: EMR 5.9.0
>Reporter: Vitaliy Savkin
>Priority: Major
> Attachments:
> _Spark_node_hanging__Thread_dump_from_application_master.txt
>
>
> One time in a month (~1000 runs) one of our spark applications freezes at
> startup. jstack says that there is a deadlock. Please see locks
> 0x802c00c0 and 0x8271bb98 in stacktraces below.
> {noformat}
> "Driver":
> at java.lang.Package.getSystemPackage(Package.java:540)
> - waiting to lock <0x802c00c0> (a java.util.HashMap)
> at java.lang.ClassLoader.getPackage(ClassLoader.java:1625)
> at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:394)
> at java.net.URLClassLoader.definePackageInternal(URLClassLoader.java:420)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:452)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> - locked <0x82789598> (a
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:221)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:210)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> - locked <0x82789540> (a
> org.apache.spark.sql.internal.NonClosableMutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:294)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:289)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2516)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405)
> - locked <0x8271bb98> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:981)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2189)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2702)
> at
> org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
> at java.net.URL.getURLStreamHandler(URL.java:1142)
> at java.net.URL.(URL.java:599)
> at java.net.URL.(URL.java:490)
> at java.net.URL.(URL.java:439)
> at java.net.JarURLConnection.parseSpecs(JarURLConnection.java:175)
> at java.net.JarURLConnection.(JarURLConnection.java:158)
> at sun.net.www.protocol.jar.JarURLConnection.(JarURLConnection.java:81)
> at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:41)
> at java.net.URL.openConnection(URL.java:979)
> at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:216)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:210)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> - locked <0x82789540> (a
> org.apache.spark.sql.internal.NonClosableMutableURLClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:262)
> at
> org.apache.spark.sql.h
[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626446#comment-16626446
]
Umayr Hassan commented on SPARK-24523:
--
[~ankur.gupta] Increasing queue size does get rid of those warning but
increasing driver core to even 8 didn't help: SparkSession is still taking many
hours to close. I'm happy to get you event logs if that would help.
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2,
> spark-stop-jstack.log.3
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624202#comment-16624202
]
Umayr Hassan edited comment on SPARK-24523 at 9/21/18 9:30 PM:
---
[~irashid] The event logs are written to HDFS.
[~ankur.gupta] Would the event logs have this information? Increasing driver
cores to 4 didn't seem to help. BTW, I also see a warning message like:
{\{18/09/21 20:15:47 WARN AsyncEventQueue: Dropped 7937 events from appStatus
since Fri Sep 21 20:14:47 UTC 2018. }}
The one thing that, for some jobs seem to help somewhat is reducing the number
of partitions. E.g. one job (that also uses LinearRegression) runs in 30min
instead of 2hrs when the number of partitions is reduced from 10 to 1 (the job
took 15min in Spark 2.0.2).
was (Author: umayr_nuna):
[~irashid] The event logs are written to HDFS.
[~ankur.gupta] Would the event logs have this information. Also, increasing
driver cores to 4 didn't seem to help. BTW, I also see a warning message like:
{{18/09/21 20:15:47 WARN AsyncEventQueue: Dropped 7937 events from appStatus
since Fri Sep 21 20:14:47 UTC 2018. }}
The one thing that, for some jobs seem to help somewhat is reducing the number
of partitions. E.g. one job (that also uses LinearRegression) runs in 30min
instead of 2hrs when the number of partitions is reduced from 10 to 1 (the job
took 15min in Spark 2.0.2).
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2,
> spark-stop-jstack.log.3
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util
[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624202#comment-16624202
]
Umayr Hassan commented on SPARK-24523:
--
[~irashid] The event logs are written to HDFS.
[~ankur.gupta] Would the event logs have this information. Also, increasing
driver cores to 4 didn't seem to help. BTW, I also see a warning message like:
{{18/09/21 20:15:47 WARN AsyncEventQueue: Dropped 7937 events from appStatus
since Fri Sep 21 20:14:47 UTC 2018. }}
The one thing that, for some jobs seem to help somewhat is reducing the number
of partitions. E.g. one job (that also uses LinearRegression) runs in 30min
instead of 2hrs when the number of partitions is reduced from 10 to 1 (the job
took 15min in Spark 2.0.2).
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2,
> spark-stop-jstack.log.3
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
>
[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622860#comment-16622860
]
Umayr Hassan commented on SPARK-24523:
--
Yet another informative stack: [^spark-stop-jstack.log.3] Especially:
{{"DataStreamer for file
/var/log/spark/apps/application_1536980808215_22630.inprogress block
BP-1050509294-10.9.48.153-1536980784099:blk_1074150491_409714" #81 daemon
prio=5 os_prio=0 tid=0x7fca79}}{{96f000 nid=0x1e1d5 in Object.wait()
[0x7fca30226000]}}{{ java.lang.Thread.State: TIMED_WAITING (on object
monitor)}}{{ at java.lang.Object.wait(Native Method)}}{{ at
org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:672)}}{{ -
locked <0x0002c3008e80> (a java.util.LinkedList)}}
Don't get why there're in-progress blocks when the application was done writing
all Parquet files more than an hour ago.
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2,
> spark-stop-jstack.log.3
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
[jira] [Updated] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-24523:
-
Attachment: spark-stop-jstack.log.3
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2,
> spark-stop-jstack.log.3
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622818#comment-16622818
]
Umayr Hassan commented on SPARK-24523:
--
Another stack trace: [^spark-stop-jstack.log.2]
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-24523:
-
Attachment: spark-stop-jstack.log.2
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1, spark-stop-jstack.log.2
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622790#comment-16622790
]
Umayr Hassan commented on SPARK-24523:
--
[~irashid] Attaching a stack trace. [^spark-stop-jstack.log.1] (I'll post
another snapshot, taken after 30min)
I see a number of "s3a-transfer-unbounded*" thread pools lying in wait. Within
the application logs, we also see a number of warning messages like
{{WARN S3AbortableInputStream: Not all bytes were read from the
S3ObjectInputStream, aborting HTTP connection. This is likely an error and may
result in sub-optimal behavior. Request only the bytes you need via a ranged
GET or drain the input stream after use. }}
These might connected, and perhaps
https://issues.apache.org/jira/browse/HADOOP-14596 is the solution. If you
agree, then maybe we can try to convince AWS EMR folks to backport the fix.
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
[jira] [Updated] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-24523:
-
Attachment: spark-stop-jstack.log.1
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
> Attachments: spark-stop-jstack.log.1
>
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24309) AsyncEventQueue should handle an interrupt from a Listener
[
https://issues.apache.org/jira/browse/SPARK-24309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621317#comment-16621317
]
Umayr Hassan commented on SPARK-24309:
--
Hi folks. I'm not sure this - or a similar - issue is resolved in 2.3.1. See
SPARK-24523. In short, we still see an exception like:
{{18/09/19 22:08:28 ERROR Utils: Uncaught exception in thread pool-4-thread-1
java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
java.lang.Thread.join(Thread.java:1252) at
java.lang.Thread.join(Thread.java:1326) at
org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:135) at
org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
at
org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1922)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1360) at
org.apache.spark.SparkContext.stop(SparkContext.scala:1921)}}
To work around this problem, we are explicitly invoking sparkSession.stop() but
that is - in some cases - causing the session to take ~2hrs to stop,
considerably increasing the job runtime. Would appreciate any thoughts here.
CC [~irashid] [~vanzin]
> AsyncEventQueue should handle an interrupt from a Listener
> --
>
> Key: SPARK-24309
> URL: https://issues.apache.org/jira/browse/SPARK-24309
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Affects Versions: 2.3.0
>Reporter: Imran Rashid
>Assignee: Imran Rashid
>Priority: Blocker
> Fix For: 2.3.1, 2.4.0
>
>
> AsyncEventQueue does not properly handle an interrupt from a Listener -- the
> spark app won't even stop!
> I observed this on an actual workload as the EventLoggingListener can
> generate an interrupt from the underlying hdfs calls:
> {noformat}
> 18/05/16 17:46:36 WARN hdfs.DFSClient: Error transferring data from
> DatanodeInfoWithStorage[10.17.206.36:20002,DS-3adac910-5d0a-418b-b0f7-6332b35bf6a1,DISK]
> to
> DatanodeInfoWithStorage[10.17.206.42:20002,DS-2e7ed0aa-0e68-441e-b5b2-96ad4a9ce7a5,DISK]:
> 10 millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.17.206.35:33950
> remote=/10.17.206.36:20002]
> 18/05/16 17:46:36 WARN hdfs.DFSClient: DataStreamer Exception
> java.net.SocketTimeoutException: 10 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/10.17.206.35:33950 remote=/10.17.206.36:20002]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2305)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$StreamerStreams.sendTransferBlock(DFSOutputStream.java:516)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1450)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1408)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1559)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1254)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:739)
> 18/05/16 17:46:36 ERROR scheduler.AsyncEventQueue: Listener
> EventLoggingListener threw an exception
> [... a few more of these ...]
> 18/05/16 17:46:36 INFO scheduler.AsyncEventQueue: Stopping listener queue
> eventLog.
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at
[jira] [Updated] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-24523:
-
Environment:
EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
was:
EMR 5.14.0, S3/HDFS inputs and outputs
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs; EMR 5.17
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-24523:
-
Affects Version/s: 2.3.1
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0, 2.3.1
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621281#comment-16621281
]
Umayr Hassan commented on SPARK-24523:
--
[~dongjoon] We still see this in Spark 2.3.1 (EMR 5.17). As a workaround, we've
been using `sparkSession.close()` but explicitly closing the session - in some
cases - is taking an additional two hours.
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24523) InterruptedException when closing SparkContext
[
https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umayr Hassan updated SPARK-24523:
-
Environment:
EMR 5.14.0, S3/HDFS inputs and outputs
was:
EMR 5.12.0, S3/HDFS inputs and outputs
> InterruptedException when closing SparkContext
> --
>
> Key: SPARK-24523
> URL: https://issues.apache.org/jira/browse/SPARK-24523
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.0
> Environment: EMR 5.14.0, S3/HDFS inputs and outputs
>
>
>
>Reporter: Umayr Hassan
>Priority: Major
>
> I'm running a Scala application in EMR with the following properties:
> {{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory
> 30g --executor-cores 5 --conf spark.default.parallelism=400 --conf
> spark.dynamicAllocation.enabled=true --conf
> spark.dynamicAllocation.maxExecutors=20 --conf
> spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
> spark.eventLog.enabled=true --conf
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
> spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
> spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
> --conf spark.yarn.maxAppAttempts=1}}
> The application runs fine till SparkContext is (automatically) closed, at
> which point the SparkContext object throws.
> {{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
> java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
> java.lang.Thread.join(Thread.java:1252) at
> java.lang.Thread.join(Thread.java:1326) at
> org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at
> org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
> org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
> org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
> org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
> at scala.util.Try$.apply(Try.scala:192) at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
>
> I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
> application), so I'm not sure which change is causing Spark 2.3 to throw. Any
> ideas?
> best,
> Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24523) InterruptedException when closing SparkContext
Umayr Hassan created SPARK-24523:
Summary: InterruptedException when closing SparkContext
Key: SPARK-24523
URL: https://issues.apache.org/jira/browse/SPARK-24523
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 2.3.0
Environment: EMR 5.12.0, S3/HDFS inputs and outputs
Reporter: Umayr Hassan
I'm running a Scala application in EMR with the following properties:
{{--master yarn --deploy-mode cluster --driver-memory 13g --executor-memory 30g
--executor-cores 5 --conf spark.default.parallelism=400 --conf
spark.dynamicAllocation.enabled=true --conf
spark.dynamicAllocation.maxExecutors=20 --conf
spark.eventLog.dir=hdfs:///var/log/spark/apps --conf
spark.eventLog.enabled=true --conf
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf
spark.scheduler.listenerbus.eventqueue.capacity=2 --conf
spark.shuffle.service.enabled=true --conf spark.sql.shuffle.partitions=400
--conf spark.yarn.maxAppAttempts=1}}
The application runs fine till SparkContext is (automatically) closed, at which
point the SparkContext object throws.
{{18/06/10 10:44:43 ERROR Utils: Uncaught exception in thread pool-4-thread-1
java.lang.InterruptedException at java.lang.Object.wait(Native Method) at
java.lang.Thread.join(Thread.java:1252) at
java.lang.Thread.join(Thread.java:1326) at
org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:133) at
org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
at
org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
at scala.collection.Iterator$class.foreach(Iterator.scala:893) at
scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219) at
org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1915)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at
org.apache.spark.SparkContext.stop(SparkContext.scala:1914) at
org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:572)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192) at
org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)}}
I've not seen this behavior in Spark 2.0.2 and Spark 2.2.0 (for the same
application), so I'm not sure which change is causing Spark 2.3 to throw. Any
ideas?
best,
Umayr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
