[jira] [Updated] (SPARK-33100) Support parse the sql statements with c-style comments

[ https://issues.apache.org/jira/browse/SPARK-33100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-33100: Description: Now the spark-sql does not support parse the sql statements with c-style coments. For exampl

[jira] [Created] (SPARK-33100) Support parse the sql statements with c-style comments

feiwang created SPARK-33100: --- Summary: Support parse the sql statements with c-style comments Key: SPARK-33100 URL: https://issues.apache.org/jira/browse/SPARK-33100 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210596#comment-17210596 ] Apache Spark commented on SPARK-33099: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210595#comment-17210595 ] Apache Spark commented on SPARK-33099: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33099: Assignee: (was: Apache Spark) > Respect executor idle timeout conf in ExecutorPodsAll

[jira] [Assigned] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

[ https://issues.apache.org/jira/browse/SPARK-33099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33099: Assignee: Apache Spark > Respect executor idle timeout conf in ExecutorPodsAllocator > --

[jira] [Created] (SPARK-33099) Respect executor idle timeout conf in ExecutorPodsAllocator

Dongjoon Hyun created SPARK-33099: - Summary: Respect executor idle timeout conf in ExecutorPodsAllocator Key: SPARK-33099 URL: https://issues.apache.org/jira/browse/SPARK-33099 Project: Spark

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Created] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

Bruce Robbins created SPARK-33098: - Summary: Exception when using 'in' to compare a partition column to a literal with the wrong type Key: SPARK-33098 URL: https://issues.apache.org/jira/browse/SPARK-33098

[jira] [Resolved] (SPARK-32743) Add distinct info at UnresolvedFunction toString

[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32743. -- Fix Version/s: 3.1.0 Assignee: ulysses you Resolution: Fixed Resolved

[jira] [Created] (SPARK-33097) Add Decommissioning messages to the Spark listener bus

Holden Karau created SPARK-33097: Summary: Add Decommissioning messages to the Spark listener bus Key: SPARK-33097 URL: https://issues.apache.org/jira/browse/SPARK-33097 Project: Spark Issue

[jira] [Commented] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210447#comment-17210447 ] Apache Spark commented on SPARK-21195: -- User 'robert3005' has created a pull reques

[jira] [Assigned] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21195: Assignee: (was: Apache Spark) > MetricSystem should pick up dynamically registered me

[jira] [Assigned] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21195: Assignee: Apache Spark > MetricSystem should pick up dynamically registered metrics in so

[jira] [Reopened] (SPARK-21195) MetricSystem should pick up dynamically registered metrics in sources

[ https://issues.apache.org/jira/browse/SPARK-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kruszewski reopened SPARK-21195: --- This has been bulk closed due to inactivity but I am still interested in getting this m

[jira] [Resolved] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33094. --- Fix Version/s: (was: 3.0.2) Resolution: Fixed Issue resolved by pull request 2997

[jira] [Assigned] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33094: - Assignee: Maxim Gekk (was: Yuning Zhang) > ORC format does not propagate Hadoop config

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210396#comment-17210396 ] Konstantinos Katsiapis commented on SPARK-22674: Does this issue only af

[jira] [Updated] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33096: -- Affects Version/s: (was: 2.3.4) > Use LinkedHashMap instead of Map for newlyCreatedExecuto

[jira] [Updated] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33096: -- Affects Version/s: 2.3.4 2.4.7 3.0.1 > Use Linke

[jira] [Resolved] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33096. --- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution: Fix

[jira] [Commented] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210355#comment-17210355 ] Apache Spark commented on SPARK-33096: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

[ https://issues.apache.org/jira/browse/SPARK-33096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33096: - Assignee: Dongjoon Hyun > Use LinkedHashMap instead of Map for newlyCreatedExecutors >

[jira] [Created] (SPARK-33096) Use LinkedHashMap instead of Map for newlyCreatedExecutors

Dongjoon Hyun created SPARK-33096: - Summary: Use LinkedHashMap instead of Map for newlyCreatedExecutors Key: SPARK-33096 URL: https://issues.apache.org/jira/browse/SPARK-33096 Project: Spark

[jira] [Commented] (SPARK-32793) Expose assert_true in Python/Scala APIs and add error message parameter

[ https://issues.apache.org/jira/browse/SPARK-32793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210269#comment-17210269 ] Apache Spark commented on SPARK-32793: -- User 'zero323' has created a pull request f

[jira] [Commented] (SPARK-32793) Expose assert_true in Python/Scala APIs and add error message parameter

[ https://issues.apache.org/jira/browse/SPARK-32793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210267#comment-17210267 ] Apache Spark commented on SPARK-32793: -- User 'zero323' has created a pull request f

[jira] [Commented] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210241#comment-17210241 ] Apache Spark commented on SPARK-33088: -- User 'fsamuel-bs' has created a pull reques

[jira] [Assigned] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33088: Assignee: (was: Apache Spark) > Enhance ExecutorPlugin API to include methods for tas

[jira] [Assigned] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33088: Assignee: Apache Spark > Enhance ExecutorPlugin API to include methods for task start and

[jira] [Commented] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210242#comment-17210242 ] Apache Spark commented on SPARK-33088: -- User 'fsamuel-bs' has created a pull reques

[jira] [Commented] (SPARK-27696) kubernetes driver pod not deleted after finish.

[ https://issues.apache.org/jira/browse/SPARK-27696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210232#comment-17210232 ] Stijn De Haes commented on SPARK-27696: --- I think it is best to have TTL controller

[jira] [Commented] (SPARK-28360) The serviceAccountName configuration item does not take effect in client mode.

[ https://issues.apache.org/jira/browse/SPARK-28360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210229#comment-17210229 ] Stijn De Haes commented on SPARK-28360: --- I think it makes sense. If you are runnin

[jira] [Updated] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33093: Labels: (was: beginner newbie) > Why do my Spark 3 jobs fail to use external shuffle service on

[jira] [Resolved] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-33093. - Resolution: Invalid > Why do my Spark 3 jobs fail to use external shuffle service on YARN? > ---

[jira] [Commented] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210186#comment-17210186 ] Yuming Wang commented on SPARK-33093: - Try to set {{spark.shuffle.useOldFetchProtoco

[jira] [Created] (SPARK-33095) Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect)

Prashant Sharma created SPARK-33095: --- Summary: Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect) Key: SPARK-33095 URL: https://issues.apache.org/jira/brows

[jira] [Commented] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210103#comment-17210103 ] Apache Spark commented on SPARK-33094: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210101#comment-17210101 ] Apache Spark commented on SPARK-33094: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33094: Assignee: Yuning Zhang (was: Apache Spark) > ORC format does not propagate Hadoop config

[jira] [Assigned] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33094: Assignee: Apache Spark (was: Yuning Zhang) > ORC format does not propagate Hadoop config

[jira] [Created] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

Maxim Gekk created SPARK-33094: -- Summary: ORC format does not propagate Hadoop config from DS options to underlying HDFS file system Key: SPARK-33094 URL: https://issues.apache.org/jira/browse/SPARK-33094

[jira] [Updated] (SPARK-33094) ORC format does not propagate Hadoop config from DS options to underlying HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-33094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-33094: --- Description: When running: {code:java} spark.read.format("orc").options(conf).load(path) {code} The

[jira] [Updated] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien updated SPARK-33093: --- Description: We are running a Spark-on-YARN setup, where each client uploads their own Spark JARs for their

[jira] [Updated] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

[ https://issues.apache.org/jira/browse/SPARK-33093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien updated SPARK-33093: --- Description: We are running a Spark-on-YARN setup, where each client uploads their own Spark JARs for their

[jira] [Created] (SPARK-33093) Why do my Spark 3 jobs fail to use external shuffle service on YARN?

Julien created SPARK-33093: -- Summary: Why do my Spark 3 jobs fail to use external shuffle service on YARN? Key: SPARK-33093 URL: https://issues.apache.org/jira/browse/SPARK-33093 Project: Spark Iss

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210072#comment-17210072 ] Jungtaek Lim commented on SPARK-28025: -- [~ste...@apache.org] Sorry to ping you on

[jira] [Updated] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

[ https://issues.apache.org/jira/browse/SPARK-32901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-32901: -- Fix Version/s: 2.4.8 > UnsafeExternalSorter may cause a SparkOutOfMemoryError to be th

[jira] [Resolved] (SPARK-33091) Avoid using map instead of foreach to avoid potential side effect at callers of OrcUtils.readCatalystSchema

[ https://issues.apache.org/jira/browse/SPARK-33091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-33091. -- Fix Version/s: 3.02 3.1.0 Assignee: Hyukjin Kwon Res