[jira] [Commented] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542113#comment-17542113 ] Sandeep Pal commented on SPARK-39283: - [~cloud_fan] This seems to occur frequently on spark 3.1.2,

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Description: We are seems this deadlock between {{TaskMemoryManager}} and

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Attachment: DeadlockSparkTasks.png > Spark tasks stuck forever due to deadlock between

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Affects Version/s: 3.1.2 (was: 3.0.0) > Spark tasks stuck forever due

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Description: We are seems this deadlock between {{TaskMemoryManager}} and

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Attachment: (was: DeadlockSparkTasks.png) > Spark tasks stuck forever due to deadlock between

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Labels: Deadlock spark3.0 (was: ) > Spark tasks stuck forever due to deadlock between

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Description: We are seems this deadlock between {{TaskMemoryManager}} and

[jira] [Created] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

Sandeep Pal created SPARK-39283: --- Summary: Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter Key: SPARK-39283 URL: https://issues.apache.org/jira/browse/SPARK-39283

[jira] [Updated] (SPARK-39283) Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-39283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-39283: Attachment: DeadlockSparkTasks.png > Spark tasks stuck forever due to deadlock between

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Affects Version/s: 1.5.0 > Spark Master shutdown automatically after some applications execution >

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Affects Version/s: (was: 1.5.1) > Spark Master shutdown automatically after some applications

[jira] [Updated] (SPARK-11534) MLLib uniformVectorRDD wrong data generation

[ https://issues.apache.org/jira/browse/SPARK-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11534: Summary: MLLib uniformVectorRDD wrong data generation (was: MLLib uniformVectorRDD wrong data

[jira] [Closed] (SPARK-11534) MLLib uniformVectorRDD wrong data generation

[ https://issues.apache.org/jira/browse/SPARK-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal closed SPARK-11534. --- Resolution: Not A Problem > MLLib uniformVectorRDD wrong data generation >

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Affects Version/s: (was: 1.5.0) > Spark Master shutdown automatically after some applications

[jira] [Created] (SPARK-11534) MLLib uniformVectorRDD wrong data distribution

Sandeep Pal created SPARK-11534: --- Summary: MLLib uniformVectorRDD wrong data distribution Key: SPARK-11534 URL: https://issues.apache.org/jira/browse/SPARK-11534 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Affects Version/s: 1.5.1 > Spark Master shutdown automatically after some applications execution >

[jira] [Updated] (SPARK-11534) MLLib uniformVectorRDD wrong data generation

[ https://issues.apache.org/jira/browse/SPARK-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11534: Description: According to the definition of uniformVectorRDD, it is supposed to generate the

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Environment: Spark Standalone on CentOS 6.6, One Master and 5 Node cluster (Worker Memory: > 150

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Environment: Spark Standalone on CentOS 6.6, One Master and 5 worker nodes cluster (Each Node

[jira] [Created] (SPARK-11487) Spark Master shutdown automatically after some applications execution

Sandeep Pal created SPARK-11487: --- Summary: Spark Master shutdown automatically after some applications execution Key: SPARK-11487 URL: https://issues.apache.org/jira/browse/SPARK-11487 Project: Spark

[jira] [Updated] (SPARK-11487) Spark Master shutdown automatically after some applications execution

[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11487: Description: The master logs are as follow after the spark automatic shutdown: 15/11/02 20:50:01

[jira] [Updated] (SPARK-11005) Spark 1.5 Shuffle performance - (sort-based shuffle)

[ https://issues.apache.org/jira/browse/SPARK-11005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11005: Summary: Spark 1.5 Shuffle performance - (sort-based shuffle) (was: Spark 1.5 Shuffle

[jira] [Created] (SPARK-11005) Spark 1.5 Shuffle performance

Sandeep Pal created SPARK-11005: --- Summary: Spark 1.5 Shuffle performance Key: SPARK-11005 URL: https://issues.apache.org/jira/browse/SPARK-11005 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-11005) Spark 1.5 Shuffle performance - (sort-based shuffle)

[ https://issues.apache.org/jira/browse/SPARK-11005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-11005: Environment: 6 node cluster with 1 master and 5 worker nodes. Memory > 100 GB each Cores = 72 each

[jira] [Commented] (SPARK-11005) Spark 1.5 Shuffle performance - (sort-based shuffle)

[ https://issues.apache.org/jira/browse/SPARK-11005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949305#comment-14949305 ] Sandeep Pal commented on SPARK-11005: - Ok, Thanks > Spark 1.5 Shuffle performance - (sort-based

[jira] [Commented] (SPARK-11005) Spark 1.5 Shuffle performance - (sort-based shuffle)

[ https://issues.apache.org/jira/browse/SPARK-11005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949117#comment-14949117 ] Sandeep Pal commented on SPARK-11005: - I think, this thread is about the different problem. Anyway, I

[jira] [Closed] (SPARK-11005) Spark 1.5 Shuffle performance - (sort-based shuffle)

[ https://issues.apache.org/jira/browse/SPARK-11005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal closed SPARK-11005. --- Resolution: Fixed Transferred to mailing list > Spark 1.5 Shuffle performance - (sort-based

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947836#comment-14947836 ] Sandeep Pal commented on SPARK-10940: - I rebooted all the machines and issue is not reproduced.

[jira] [Closed] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal closed SPARK-10940. --- Resolution: Fixed Could not able to reproduce after restarting all the machines. > Too many open

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945449#comment-14945449 ] Sandeep Pal commented on SPARK-10940: - Ok sure, how much do you suggest to increase further. > Too

[jira] [Updated] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-10940: Description: Executing terasort by Spark-SQL on the data generated by teragen in hadoop. Data

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945652#comment-14945652 ] Sandeep Pal commented on SPARK-10940: - Ok, I will. One more important think I for got to mention. The

[jira] [Comment Edited] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945652#comment-14945652 ] Sandeep Pal edited comment on SPARK-10940 at 10/6/15 7:55 PM: -- Ok, I will.

[jira] [Comment Edited] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945652#comment-14945652 ] Sandeep Pal edited comment on SPARK-10940 at 10/6/15 7:54 PM: -- Ok, I will.

[jira] [Closed] (SPARK-9040) StructField datatype Conversion Error

[ https://issues.apache.org/jira/browse/SPARK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal closed SPARK-9040. -- Resolution: Fixed Fix Version/s: 1.4.0 > StructField datatype Conversion Error >

[jira] [Comment Edited] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945390#comment-14945390 ] Sandeep Pal edited comment on SPARK-10940 at 10/6/15 5:30 PM: -- Yes, I am

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945390#comment-14945390 ] Sandeep Pal commented on SPARK-10940: - Yes, I am using the default shuffle manager in spark 1.5 which

[jira] [Updated] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-10940: Description: Executing terasort by Spark-SQL on the data generated by teragen in hadoop. Data

[jira] [Created] (SPARK-10940) Too many open files Spark Shuffle

Sandeep Pal created SPARK-10940: --- Summary: Too many open files Spark Shuffle Key: SPARK-10940 URL: https://issues.apache.org/jira/browse/SPARK-10940 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-10940) Too many open files Spark Shuffle

[ https://issues.apache.org/jira/browse/SPARK-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944307#comment-14944307 ] Sandeep Pal commented on SPARK-10940: - I have already read similar issue as below but this does not

[jira] [Reopened] (SPARK-9282) Filter on Spark DataFrame with multiple columns

[ https://issues.apache.org/jira/browse/SPARK-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal reopened SPARK-9282: On using '' instead of 'and', the following error occurs: Py4JError

[jira] [Updated] (SPARK-9040) StructField datatype Conversion Error

[ https://issues.apache.org/jira/browse/SPARK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-9040: --- Description: The following issue occurs if I specify the StructFields in specific order in

[jira] [Updated] (SPARK-9282) Join on Spark DataFrame with multiple columns

[ https://issues.apache.org/jira/browse/SPARK-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-9282: --- Description: Filter on dataframe does not work if we have more than one column inside the filter.

[jira] [Created] (SPARK-9283) Join on Spark DataFrame with multiple columns

Sandeep Pal created SPARK-9283: -- Summary: Join on Spark DataFrame with multiple columns Key: SPARK-9283 URL: https://issues.apache.org/jira/browse/SPARK-9283 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-9282) Filter on Spark DataFrame with multiple columns

[ https://issues.apache.org/jira/browse/SPARK-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-9282: --- Summary: Filter on Spark DataFrame with multiple columns (was: Join on Spark DataFrame with multiple

[jira] [Created] (SPARK-9282) Join on Spark DataFrame with multiple columns

Sandeep Pal created SPARK-9282: -- Summary: Join on Spark DataFrame with multiple columns Key: SPARK-9282 URL: https://issues.apache.org/jira/browse/SPARK-9282 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-9040) StructField datatype Conversion Error

[ https://issues.apache.org/jira/browse/SPARK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-9040: --- Component/s: Spark Core StructField datatype Conversion Error -

[jira] [Updated] (SPARK-9040) StructField datatype Conversion Error

[ https://issues.apache.org/jira/browse/SPARK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated SPARK-9040: --- Component/s: PySpark StructField datatype Conversion Error -

[jira] [Created] (SPARK-9040) StructField datatype Conversion Error

Sandeep Pal created SPARK-9040: -- Summary: StructField datatype Conversion Error Key: SPARK-9040 URL: https://issues.apache.org/jira/browse/SPARK-9040 Project: Spark Issue Type: Bug Affects