[jira] [Updated] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads
[
https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksii Kostyliev updated SPARK-7233:
-
Attachment: blocked_threads_closurecleaner.png
> ClosureCleaner#clean blocks concurrent job submitter threads
>
>
> Key: SPARK-7233
> URL: https://issues.apache.org/jira/browse/SPARK-7233
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Oleksii Kostyliev
> Attachments: blocked_threads_closurecleaner.png
>
>
> {{org.apache.spark.util.ClosureCleaner#clean}} method contains logic to
> determine if Spark is run in interpreter mode:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L120
> While this behavior is indeed valuable in particular situations, in addition
> to this it causes concurrent submitter threads to be blocked on a native call
> to {{java.lang.Class#forName0}} since it appears only 1 thread at a time can
> make the call.
> This becomes a major issue when you have multiple threads concurrently
> submitting short-lived jobs. This is one of the patterns how we use Spark in
> production, and the number of parallel requests is expected to be quite high,
> up to a couple of thousand at a time.
> A typical stacktrace of a blocked thread looks like:
> {code}
> http-bio-8091-exec-14 [BLOCKED] [DAEMON]
> java.lang.Class.forName0(String, boolean, ClassLoader, Class) Class.java
> (native)
> java.lang.Class.forName(String) Class.java:260
> org.apache.spark.util.ClosureCleaner$.clean(Object, boolean)
> ClosureCleaner.scala:122
> org.apache.spark.SparkContext.clean(Object, boolean) SparkContext.scala:1623
> org.apache.spark.rdd.RDD.reduce(Function2) RDD.scala:883
> org.apache.spark.rdd.RDD.takeOrdered(int, Ordering) RDD.scala:1240
> org.apache.spark.api.java.JavaRDDLike$class.takeOrdered(JavaRDDLike, int,
> Comparator) JavaRDDLike.scala:586
> org.apache.spark.api.java.AbstractJavaRDDLike.takeOrdered(int, Comparator)
> JavaRDDLike.scala:46
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads
[
https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-7233:
---
Priority: Critical (was: Major)
> ClosureCleaner#clean blocks concurrent job submitter threads
>
>
> Key: SPARK-7233
> URL: https://issues.apache.org/jira/browse/SPARK-7233
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Oleksii Kostyliev
>Priority: Critical
> Attachments: blocked_threads_closurecleaner.png
>
>
> {{org.apache.spark.util.ClosureCleaner#clean}} method contains logic to
> determine if Spark is run in interpreter mode:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L120
> While this behavior is indeed valuable in particular situations, in addition
> to this it causes concurrent submitter threads to be blocked on a native call
> to {{java.lang.Class#forName0}} since it appears only 1 thread at a time can
> make the call.
> This becomes a major issue when you have multiple threads concurrently
> submitting short-lived jobs. This is one of the patterns how we use Spark in
> production, and the number of parallel requests is expected to be quite high,
> up to a couple of thousand at a time.
> A typical stacktrace of a blocked thread looks like:
> {code}
> http-bio-8091-exec-14 [BLOCKED] [DAEMON]
> java.lang.Class.forName0(String, boolean, ClassLoader, Class) Class.java
> (native)
> java.lang.Class.forName(String) Class.java:260
> org.apache.spark.util.ClosureCleaner$.clean(Object, boolean)
> ClosureCleaner.scala:122
> org.apache.spark.SparkContext.clean(Object, boolean) SparkContext.scala:1623
> org.apache.spark.rdd.RDD.reduce(Function2) RDD.scala:883
> org.apache.spark.rdd.RDD.takeOrdered(int, Ordering) RDD.scala:1240
> org.apache.spark.api.java.JavaRDDLike$class.takeOrdered(JavaRDDLike, int,
> Comparator) JavaRDDLike.scala:586
> org.apache.spark.api.java.AbstractJavaRDDLike.takeOrdered(int, Comparator)
> JavaRDDLike.scala:46
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads
[
https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-7233:
---
Target Version/s: 1.4.0
> ClosureCleaner#clean blocks concurrent job submitter threads
>
>
> Key: SPARK-7233
> URL: https://issues.apache.org/jira/browse/SPARK-7233
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Oleksii Kostyliev
>Priority: Critical
> Attachments: blocked_threads_closurecleaner.png
>
>
> {{org.apache.spark.util.ClosureCleaner#clean}} method contains logic to
> determine if Spark is run in interpreter mode:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L120
> While this behavior is indeed valuable in particular situations, in addition
> to this it causes concurrent submitter threads to be blocked on a native call
> to {{java.lang.Class#forName0}} since it appears only 1 thread at a time can
> make the call.
> This becomes a major issue when you have multiple threads concurrently
> submitting short-lived jobs. This is one of the patterns how we use Spark in
> production, and the number of parallel requests is expected to be quite high,
> up to a couple of thousand at a time.
> A typical stacktrace of a blocked thread looks like:
> {code}
> http-bio-8091-exec-14 [BLOCKED] [DAEMON]
> java.lang.Class.forName0(String, boolean, ClassLoader, Class) Class.java
> (native)
> java.lang.Class.forName(String) Class.java:260
> org.apache.spark.util.ClosureCleaner$.clean(Object, boolean)
> ClosureCleaner.scala:122
> org.apache.spark.SparkContext.clean(Object, boolean) SparkContext.scala:1623
> org.apache.spark.rdd.RDD.reduce(Function2) RDD.scala:883
> org.apache.spark.rdd.RDD.takeOrdered(int, Ordering) RDD.scala:1240
> org.apache.spark.api.java.JavaRDDLike$class.takeOrdered(JavaRDDLike, int,
> Comparator) JavaRDDLike.scala:586
> org.apache.spark.api.java.AbstractJavaRDDLike.takeOrdered(int, Comparator)
> JavaRDDLike.scala:46
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads
[
https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-7233:
-
Fix Version/s: 1.4.0
> ClosureCleaner#clean blocks concurrent job submitter threads
>
>
> Key: SPARK-7233
> URL: https://issues.apache.org/jira/browse/SPARK-7233
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Oleksii Kostyliev
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: blocked_threads_closurecleaner.png
>
>
> {{org.apache.spark.util.ClosureCleaner#clean}} method contains logic to
> determine if Spark is run in interpreter mode:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L120
> While this behavior is indeed valuable in particular situations, in addition
> to this it causes concurrent submitter threads to be blocked on a native call
> to {{java.lang.Class#forName0}} since it appears only 1 thread at a time can
> make the call.
> This becomes a major issue when you have multiple threads concurrently
> submitting short-lived jobs. This is one of the patterns how we use Spark in
> production, and the number of parallel requests is expected to be quite high,
> up to a couple of thousand at a time.
> A typical stacktrace of a blocked thread looks like:
> {code}
> http-bio-8091-exec-14 [BLOCKED] [DAEMON]
> java.lang.Class.forName0(String, boolean, ClassLoader, Class) Class.java
> (native)
> java.lang.Class.forName(String) Class.java:260
> org.apache.spark.util.ClosureCleaner$.clean(Object, boolean)
> ClosureCleaner.scala:122
> org.apache.spark.SparkContext.clean(Object, boolean) SparkContext.scala:1623
> org.apache.spark.rdd.RDD.reduce(Function2) RDD.scala:883
> org.apache.spark.rdd.RDD.takeOrdered(int, Ordering) RDD.scala:1240
> org.apache.spark.api.java.JavaRDDLike$class.takeOrdered(JavaRDDLike, int,
> Comparator) JavaRDDLike.scala:586
> org.apache.spark.api.java.AbstractJavaRDDLike.takeOrdered(int, Comparator)
> JavaRDDLike.scala:46
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
