[jira] [Commented] (SPARK-38536) Spark 3 can not read mixed format partitions

[ https://issues.apache.org/jira/browse/SPARK-38536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17506728#comment-17506728 ] Deegue commented on SPARK-38536: I wonder if it's caused by the different Hadoop version

[jira] [Commented] (SPARK-38536) Spark 3 can not read mixed format partitions

[ https://issues.apache.org/jira/browse/SPARK-38536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17506678#comment-17506678 ] Deegue commented on SPARK-38536: Thanks [~hyukjin.kwon], [~songhuicheng] I don't think t

[jira] [Created] (SPARK-29910) Add minimum runtime limit to speculation

Deegue created SPARK-29910: -- Summary: Add minimum runtime limit to speculation Key: SPARK-29910 URL: https://issues.apache.org/jira/browse/SPARK-29910 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-29786) Fix MetaException when dropping a partition not exists on HDFS.

Deegue created SPARK-29786: -- Summary: Fix MetaException when dropping a partition not exists on HDFS. Key: SPARK-29786 URL: https://issues.apache.org/jira/browse/SPARK-29786 Project: Spark Issue Ty

[jira] [Created] (SPARK-29785) Optimize opening a new session of Spark Thrift Server

Deegue created SPARK-29785: -- Summary: Optimize opening a new session of Spark Thrift Server Key: SPARK-29785 URL: https://issues.apache.org/jira/browse/SPARK-29785 Project: Spark Issue Type: Improve

[jira] [Updated] (SPARK-28239) Allow TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Summary: Allow TCP connections created by shuffle service auto close on YARN NodeManagers (was: Make TCP co

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Environment: Hadoop2.6.0-CDH5.8.3(netty3) Spark2.4.0(netty4) Configs: spark.shuffle.service.enabled=true

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Attachment: screenshot-2.png > Make TCP connections created by shuffle service auto close on YARN > NodeMan

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Attachment: (was: screenshot-1.png) > Make TCP connections created by shuffle service auto close on YARN

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Attachment: screenshot-1.png > Make TCP connections created by shuffle service auto close on YARN > NodeMan

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Attachment: screenshot-1.png > Make TCP connections created by shuffle service auto close on YARN > NodeMan

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When executing shuffle tasks, (was: When we set spark.shuffle.service.enabled=true, ) > Make

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Description: When we set spark.shuffle.service.enabled=true, > Make TCP connections created by shuffle serv

[jira] [Updated] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

[ https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-28239: --- Environment: Hadoop2.6.0-CDH5.8.3(netty3) Spark2.4.0(netty4) set spark.shuffle.service.enabled=true was:

[jira] [Created] (SPARK-28239) Make TCP connections created by shuffle service auto close on YARN NodeManagers

Deegue created SPARK-28239: -- Summary: Make TCP connections created by shuffle service auto close on YARN NodeManagers Key: SPARK-28239 URL: https://issues.apache.org/jira/browse/SPARK-28239 Project: Spark

[jira] [Updated] (SPARK-26667) Add `Scanning Input Table` to Performance Tuning Guide

[ https://issues.apache.org/jira/browse/SPARK-26667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-26667: --- Description: We can use `CombineTextInputFormat` instead of `TextInputFormat` and set configurations to inc

[jira] [Created] (SPARK-26667) Add `Scanning Input Table` to Performance Tuning Guide

Deegue created SPARK-26667: -- Summary: Add `Scanning Input Table` to Performance Tuning Guide Key: SPARK-26667 URL: https://issues.apache.org/jira/browse/SPARK-26667 Project: Spark Issue Type: Docume

[jira] [Commented] (SPARK-26630) ClassCastException in TableReader while creating HadoopRDD

[ https://issues.apache.org/jira/browse/SPARK-26630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746985#comment-16746985 ] Deegue commented on SPARK-26630: Hi [~dongjoon], It seems all of the versions are affec

[jira] [Updated] (SPARK-26630) ClassCastException in TableReader while creating HadoopRDD

[ https://issues.apache.org/jira/browse/SPARK-26630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-26630: --- Affects Version/s: 2.4.1 2.4.0 > ClassCastException in TableReader while creating Had

[jira] [Created] (SPARK-26630) ClassCastException in TableReader while creating HadoopRDD

Deegue created SPARK-26630: -- Summary: ClassCastException in TableReader while creating HadoopRDD Key: SPARK-26630 URL: https://issues.apache.org/jira/browse/SPARK-26630 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-26577) Add input optimizer when reading Hive table by SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-26577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-26577: --- Description: When using SparkSQL, for example the ThriftServer, if we set `spark.sql.hive.inputFormat.optim

[jira] [Updated] (SPARK-26577) Add input optimizer when reading Hive table by SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-26577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-26577: --- External issue URL: https://github.com/apache/spark/pull/23496 > Add input optimizer when reading Hive table

[jira] [Created] (SPARK-26577) Add input optimizer when reading Hive table by SparkSQL

Deegue created SPARK-26577: -- Summary: Add input optimizer when reading Hive table by SparkSQL Key: SPARK-26577 URL: https://issues.apache.org/jira/browse/SPARK-26577 Project: Spark Issue Type: Impro

[jira] [Reopened] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue reopened SPARK-24672: Coditions to this issue: When the amount of data I selected is larger than spark.driver.maxResultSize , It r

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Attachment: image4.png > No job is running but there are active tasks >

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Description: Issues:   There are active tasks while no job is running. These active tasks occupy the exec

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Attachment: image3.png image2.png > No job is running but there are active tasks > -

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Description: Issues:   There are active tasks while no job is running. These active tasks occupy the exec

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Environment: hadoop 2.6.0 spark 2.2.1 CDH 5.8.3 java 1.8.0   More info : image1.png in Attachments

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Attachment: image1.png > No job is running but there are active tasks >

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Environment: hadoop 2.6.0 spark 2.2.1 CDH 5.8.3 java 1.8.0   image1.png was: hadoop 2.6.0 spark 2.2

[jira] [Created] (SPARK-24672) No job is running but there are active tasks

Deegue created SPARK-24672: -- Summary: No job is running but there are active tasks Key: SPARK-24672 URL: https://issues.apache.org/jira/browse/SPARK-24672 Project: Spark Issue Type: Bug Co