[jira] [Commented] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210596#comment-17210596 ] Apache Spark commented on SPARK-33099: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210595#comment-17210595 ] Apache Spark commented on SPARK-33099: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33099: Assignee: (was: Apache Spark) > Respect executor idle timeout conf in

[jira] [Assigned] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33099: Assignee: Apache Spark > Respect executor idle timeout conf in ExecutorPodsAllocator >

[jira] [Created] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

Dongjoon Hyun created SPARK-33099: - Summary: Respect executor idle timeout conf in ExecutorPodsAllocator Key: SPARK-33099 URL: https://issues.apache.org/jira/browse/SPARK-33099 Project: Spark

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Created] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

Bruce Robbins created SPARK-33098: - Summary: Exception when using 'in' to compare a partition column to a literal with the wrong type Key: SPARK-33098 URL: https://issues.apache.org/jira/browse/SPARK-33098

[jira] [Resolved] (SPARK-32743) Add distinct info at UnresolvedFunction toString

[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32743. -- Fix Version/s: 3.1.0 Assignee: ulysses you Resolution: Fixed Resolved

[jira] [Created] (SPARK-33097) Add Decommissioning messages to the Spark listener bus

Holden Karau created SPARK-33097: Summary: Add Decommissioning messages to the Spark listener bus Key: SPARK-33097 URL: https://issues.apache.org/jira/browse/SPARK-33097 Project: Spark Issue

[jira] [Commented] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210447#comment-17210447 ] Apache Spark commented on SPARK-21195: -- User 'robert3005' has created a pull request for this

[jira] [Assigned] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21195: Assignee: (was: Apache Spark) > MetricSystem should pick up dynamically registered

[jira] [Assigned] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21195: Assignee: Apache Spark > MetricSystem should pick up dynamically registered metrics in

[jira] [Reopened] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kruszewski reopened SPARK-21195: --- This has been bulk closed due to inactivity but I am still interested in getting this

[jira] [Resolved] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33094. --- Fix Version/s: (was: 3.0.2) Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33094: - Assignee: Maxim Gekk (was: Yuning Zhang) > ORC format does not propagate Hadoop

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210396#comment-17210396 ] Konstantinos Katsiapis commented on SPARK-22674: Does this issue only affect PySpark2 or

[jira] [Updated] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33096: -- Affects Version/s: (was: 2.3.4) > Use LinkedHashMap instead of Map for

[jira] [Updated] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33096: -- Affects Version/s: 2.3.4 2.4.7 3.0.1 > Use

[jira] [Resolved] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33096. --- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution:

[jira] [Commented] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210355#comment-17210355 ] Apache Spark commented on SPARK-33096: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33096: - Assignee: Dongjoon Hyun > Use LinkedHashMap instead of Map for newlyCreatedExecutors >

[jira] [Created] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

Dongjoon Hyun created SPARK-33096: - Summary: Use LinkedHashMap instead of Map for newlyCreatedExecutors Key: SPARK-33096 URL: https://issues.apache.org/jira/browse/SPARK-33096 Project: Spark

[jira] [Commented] (SPARK-32793) Expose assert_true in Python/Scala APIs and add error message parameter

[ https://issues.apache.org/jira/browse/SPARK-32793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210269#comment-17210269 ] Apache Spark commented on SPARK-32793: -- User 'zero323' has created a pull request for this issue:

[jira] [Commented] (SPARK-32793) Expose assert_true in Python/Scala APIs and add error message parameter

[ https://issues.apache.org/jira/browse/SPARK-32793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210267#comment-17210267 ] Apache Spark commented on SPARK-32793: -- User 'zero323' has created a pull request for this issue:

[jira] [Commented] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210241#comment-17210241 ] Apache Spark commented on SPARK-33088: -- User 'fsamuel-bs' has created a pull request for this

[jira] [Assigned] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33088: Assignee: (was: Apache Spark) > Enhance ExecutorPlugin API to include methods for

[jira] [Assigned] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33088: Assignee: Apache Spark > Enhance ExecutorPlugin API to include methods for task start

[jira] [Commented] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210242#comment-17210242 ] Apache Spark commented on SPARK-33088: -- User 'fsamuel-bs' has created a pull request for this

[jira] [Commented] (SPARK-27696) kubernetes driver pod not deleted after finish.

[ https://issues.apache.org/jira/browse/SPARK-27696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210232#comment-17210232 ] Stijn De Haes commented on SPARK-27696: --- I think it is best to have TTL controller for finished

[jira] [Commented] (SPARK-28360) The serviceAccountName configuration item does not take effect in client mode.

[ https://issues.apache.org/jira/browse/SPARK-28360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210229#comment-17210229 ] Stijn De Haes commented on SPARK-28360: --- I think it makes sense. If you are running in client mode

[jira] [Updated] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33093: Labels: (was: beginner newbie) > Why do my Spark 3 jobs fail to use external shuffle service on

[jira] [Resolved] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-33093. - Resolution: Invalid > Why do my Spark 3 jobs fail to use external shuffle service on YARN? >

[jira] [Commented] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210186#comment-17210186 ] Yuming Wang commented on SPARK-33093: - Try to set {{spark.shuffle.useOldFetchProtocol}} to {{true}}.

[jira] [Created] (SPARK-33095) Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect)

Prashant Sharma created SPARK-33095: --- Summary: Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect) Key: SPARK-33095 URL:

[jira] [Commented] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210103#comment-17210103 ] Apache Spark commented on SPARK-33094: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210101#comment-17210101 ] Apache Spark commented on SPARK-33094: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33094: Assignee: Yuning Zhang (was: Apache Spark) > ORC format does not propagate Hadoop

[jira] [Assigned] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33094: Assignee: Apache Spark (was: Yuning Zhang) > ORC format does not propagate Hadoop

[jira] [Created] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

Maxim Gekk created SPARK-33094: -- Summary: ORC format does not propagate Hadoop config from DS options to underlying HDFS file system Key: SPARK-33094 URL: https://issues.apache.org/jira/browse/SPARK-33094

[jira] [Updated] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-33094: --- Description: When running: {code:java} spark.read.format("orc").options(conf).load(path) {code} The

[jira] [Updated] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien updated SPARK-33093: --- Description: We are running a Spark-on-YARN setup, where each client uploads their own Spark JARs for

[jira] [Updated] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien updated SPARK-33093: --- Description: We are running a Spark-on-YARN setup, where each client uploads their own Spark JARs for

[jira] [Created] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

Julien created SPARK-33093: -- Summary: Why do my Spark 3 jobs fail to use external shuffle service on YARN? Key: SPARK-33093 URL: https://issues.apache.org/jira/browse/SPARK-33093 Project: Spark

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210072#comment-17210072 ] Jungtaek Lim commented on SPARK-28025: -- [~ste...@apache.org] Sorry to ping you on the old issue.

[jira] [Updated] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

[ https://issues.apache.org/jira/browse/SPARK-32901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-32901: -- Fix Version/s: 2.4.8 > UnsafeExternalSorter may cause a SparkOutOfMemoryError to be

[jira] [Resolved] (SPARK-33091) Avoid using map instead of foreach to avoid potential side effect at callers of OrcUtils.readCatalystSchema

[ https://issues.apache.org/jira/browse/SPARK-33091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-33091. -- Fix Version/s: 3.02 3.1.0 Assignee: Hyukjin Kwon

[jira] [Assigned] (SPARK-33092) Support subexpression elimination in ProjectExec

[ https://issues.apache.org/jira/browse/SPARK-33092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-33092: --- Assignee: L. C. Hsieh (was: Apache Spark) > Support subexpression elimination in

[jira] [Assigned] (SPARK-33092) Support subexpression elimination in ProjectExec

[ https://issues.apache.org/jira/browse/SPARK-33092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33092: Assignee: Apache Spark (was: L. C. Hsieh) > Support subexpression elimination in

[jira] [Assigned] (SPARK-33092) Support subexpression elimination in ProjectExec

[ https://issues.apache.org/jira/browse/SPARK-33092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33092: Assignee: Apache Spark (was: L. C. Hsieh) > Support subexpression elimination in

[jira] [Assigned] (SPARK-33092) Support subexpression elimination in ProjectExec

[ https://issues.apache.org/jira/browse/SPARK-33092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33092: Assignee: L. C. Hsieh (was: Apache Spark) > Support subexpression elimination in

[jira] [Commented] (SPARK-33092) Support subexpression elimination in ProjectExec

[ https://issues.apache.org/jira/browse/SPARK-33092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210024#comment-17210024 ] Apache Spark commented on SPARK-33092: -- User 'viirya' has created a pull request for this issue: