[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Description: h2. Instruction This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay. !screenshot-2.png! Also, you may find that these hanging tasks belong to the same executors. Usually, in this case, you will also get nothing helpful from the executor log. If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK-8194653 issue. !screenshot-1.png! h2. Solutions Here are some options to circumvent this problem: 1. For the cluster managers side, you can update the JDK version according to https://bugs.openjdk.java.net/browse/JDK-8194653 2. If you are not able to update the JDK version for the cluster entirely, you can use `spark.executorEnv.JAVA_HOME` to specify a suitable JRE for your apps 2. Also, turn on `spark.speculation` may let spark automatically re-run the hanging tasks and bypass the problem was: h2. Instruction This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay. !screenshot-2.png! Also, you may find that these hanging tasks belong to the same executors. Usually, in this case, you will also get nothing helpful from the executor log. If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK-8194653 issue. !screenshot-1.png! h2. Solutions 1. Update JDK version > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png, screenshot-2.png > > > h2. Instruction > This will cause deadlock and hangs concurrent tasks forever on the same > executor. for example, > > In the Spark UI stage tab, you may find some of the tasks hang for hours and > all others complete without delay. > > !screenshot-2.png! > Also, you may find that these hanging tasks belong to the same executors. > Usually, in this case, you will also get nothing helpful from the executor > log. > If you print the executor jstack or you check the ThreadDump via SparkUI > executor tab and you find some task thread blocked like below, you are very > likely to hit the JDK-8194653 issue. > !screenshot-1.png! > h2. Solutions > Here are some options to circumvent this problem: > 1. For the cluster managers side, you can update the JDK version according to > https://bugs.openjdk.java.net/browse/JDK-8194653 > 2. If you are not able to update the JDK version for the cluster entirely, > you can use `spark.executorEnv.JAVA_HOME` to specify a suitable JRE for your > apps > 2. Also, turn on `spark.speculation` may let spark automatically re-run the > hanging tasks and bypass the problem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Description: h2. Instruction This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay. !screenshot-2.png! Also, you may find that these hanging tasks belong to the same executors. Usually, in this case, you will also get nothing helpful from the executor log. If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK-8194653 issue. !screenshot-1.png! h2. Solutions 1. Update JDK version was: This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay. !screenshot-2.png! Also, you may find that these hanging tasks belong to the same executors. Usually, in this case, you will also get nothing helpful from the executor log. If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK !screenshot-1.png! > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png, screenshot-2.png > > > h2. Instruction > This will cause deadlock and hangs concurrent tasks forever on the same > executor. for example, > > In the Spark UI stage tab, you may find some of the tasks hang for hours and > all others complete without delay. > > !screenshot-2.png! > Also, you may find that these hanging tasks belong to the same executors. > Usually, in this case, you will also get nothing helpful from the executor > log. > If you print the executor jstack or you check the ThreadDump via SparkUI > executor tab and you find some task thread blocked like below, you are very > likely to hit the JDK-8194653 issue. > !screenshot-1.png! > h2. Solutions > 1. Update JDK version -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Description: This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay. !screenshot-2.png! Also, you may find that these hanging tasks belong to the same executors. Usually, in this case, you will also get nothing helpful from the executor log. If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK !screenshot-1.png! was: This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some task hang for hours !screenshot-2.png! !screenshot-1.png! > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png, screenshot-2.png > > > This will cause deadlock and hangs concurrent tasks forever on the same > executor. for example, > > In the Spark UI stage tab, you may find some of the tasks hang for hours and > all others complete without delay. > > !screenshot-2.png! > Also, you may find that these hanging tasks belong to the same executors. > Usually, in this case, you will also get nothing helpful from the executor > log. > If you print the executor jstack or you check the ThreadDump via SparkUI > executor tab and you find some task thread blocked like below, you are very > likely to hit the JDK > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Description: This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, In the Spark UI stage tab, you may find some task hang for hours !screenshot-2.png! !screenshot-1.png! was: This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, !screenshot-1.png! > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png, screenshot-2.png > > > This will cause deadlock and hangs concurrent tasks forever on the same > executor. for example, > > In the Spark UI stage tab, you may find some task hang for hours > !screenshot-2.png! > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Attachment: screenshot-2.png > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png, screenshot-2.png > > > This will cause deadlock and hangs concurrent tasks forever on the same > executor. for example, > > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Description: This will cause deadlock and hangs concurrent tasks forever on the same executor. for example, !screenshot-1.png! > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png > > > This will cause deadlock and hangs concurrent tasks forever on the same > executor. for example, > > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Attachment: screenshot-1.png > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log, screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34523) JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call
[ https://issues.apache.org/jira/browse/SPARK-34523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-34523: - Summary: JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call (was: JDK-8194653: JDK-8194653 Deadlock involving FileSystems.getDefault and System.loadLibrary call) > JDK-8194653: Deadlock involving FileSystems.getDefault and > System.loadLibrary call > --- > > Key: SPARK-34523 > URL: https://issues.apache.org/jira/browse/SPARK-34523 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 2.4.7, 3.0.2, 3.1.1 >Reporter: Kent Yao >Priority: Major > Attachments: 4303.log > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
