[jira] [Created] (SPARK-43446) Upgrade Apache Arrow to 12.0.0

Dongjoon Hyun created SPARK-43446: - Summary: Upgrade Apache Arrow to 12.0.0 Key: SPARK-43446 URL: https://issues.apache.org/jira/browse/SPARK-43446 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-43424) Support vanila JDBC CHAR/VARCHAR through STS

[ https://issues.apache.org/jira/browse/SPARK-43424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43424. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41102

[jira] [Assigned] (SPARK-43424) Support vanila JDBC CHAR/VARCHAR through STS

[ https://issues.apache.org/jira/browse/SPARK-43424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43424: - Assignee: Kent Yao > Support vanila JDBC CHAR/VARCHAR through STS >

[jira] [Created] (SPARK-43445) Enable GroupBySlowTests.test_split_apply_combine_on_series for pandas 2.0.0.

Haejoon Lee created SPARK-43445: --- Summary: Enable GroupBySlowTests.test_split_apply_combine_on_series for pandas 2.0.0. Key: SPARK-43445 URL: https://issues.apache.org/jira/browse/SPARK-43445 Project:

[jira] [Created] (SPARK-43444) Enable GroupBySlowTests.test_value_counts for pandas 2.0.0.

Haejoon Lee created SPARK-43444: --- Summary: Enable GroupBySlowTests.test_value_counts for pandas 2.0.0. Key: SPARK-43444 URL: https://issues.apache.org/jira/browse/SPARK-43444 Project: Spark

[jira] [Resolved] (SPARK-43441) makeDotNode should not fail when DeterministicLevel is absent

[ https://issues.apache.org/jira/browse/SPARK-43441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43441. --- Fix Version/s: 3.4.1 Resolution: Fixed Issue resolved by pull request 41124

[jira] [Created] (SPARK-43443) Add benchmark for Timestamp type inference when use invalid value

Jia Fan created SPARK-43443: --- Summary: Add benchmark for Timestamp type inference when use invalid value Key: SPARK-43443 URL: https://issues.apache.org/jira/browse/SPARK-43443 Project: Spark

[jira] [Commented] (SPARK-43442) Split test module `pyspark_pandas_connect`

[ https://issues.apache.org/jira/browse/SPARK-43442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721609#comment-17721609 ] Snoot.io commented on SPARK-43442: -- User 'zhengruifeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-43442) Split test module `pyspark_pandas_connect`

[ https://issues.apache.org/jira/browse/SPARK-43442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721610#comment-17721610 ] Snoot.io commented on SPARK-43442: -- User 'zhengruifeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-43403) GET /history//1/jobs/ failed: java.lang.IllegalStateException: DB is closed

[ https://issues.apache.org/jira/browse/SPARK-43403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721605#comment-17721605 ] Snoot.io commented on SPARK-43403: -- User 'zhouyifan279' has created a pull request for this issue:

[jira] [Created] (SPARK-43442) Split test module `pyspark_pandas_connect`

Ruifeng Zheng created SPARK-43442: - Summary: Split test module `pyspark_pandas_connect` Key: SPARK-43442 URL: https://issues.apache.org/jira/browse/SPARK-43442 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43425) Add TimestampNTZType to ColumnarBatchRow

[ https://issues.apache.org/jira/browse/SPARK-43425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43425: - Fix Version/s: 3.4.1 > Add TimestampNTZType to ColumnarBatchRow >

[jira] [Updated] (SPARK-43440) Support registration of an Arrow-optimized Python UDF

[ https://issues.apache.org/jira/browse/SPARK-43440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43440: - Description: Currently, when users register an Arrow-optimized Python UDF, it will be

[jira] [Assigned] (SPARK-43441) makeDotNode should not fail when DeterministicLevel is absent

[ https://issues.apache.org/jira/browse/SPARK-43441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43441: - Assignee: Qi Tan > makeDotNode should not fail when DeterministicLevel is absent >

[jira] [Created] (SPARK-43441) makeDotNode should not fail when DeterministicLevel is absent

Qi Tan created SPARK-43441: -- Summary: makeDotNode should not fail when DeterministicLevel is absent Key: SPARK-43441 URL: https://issues.apache.org/jira/browse/SPARK-43441 Project: Spark Issue

[jira] [Created] (SPARK-43440) Support registration of an Arrow-optimized Python UDF

Xinrong Meng created SPARK-43440: Summary: Support registration of an Arrow-optimized Python UDF Key: SPARK-43440 URL: https://issues.apache.org/jira/browse/SPARK-43440 Project: Spark Issue

[jira] [Commented] (SPARK-42523) Apache Spark 3.4 release

[ https://issues.apache.org/jira/browse/SPARK-42523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721549#comment-17721549 ] Xinrong Meng commented on SPARK-42523: -- I am wondering if we shall keep the ticket open for minor

[jira] [Resolved] (SPARK-43412) Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow-optimized Python UDFs

[ https://issues.apache.org/jira/browse/SPARK-43412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-43412. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41053

[jira] [Resolved] (SPARK-43430) ExecutePlanRequest should have the ability to set request options.

[ https://issues.apache.org/jira/browse/SPARK-43430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-43430. --- Fix Version/s: 3.5.0 Resolution: Fixed > ExecutePlanRequest should have the

[jira] [Assigned] (SPARK-43430) ExecutePlanRequest should have the ability to set request options.

[ https://issues.apache.org/jira/browse/SPARK-43430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-43430: - Assignee: Martin Grund > ExecutePlanRequest should have the ability to set

[jira] [Updated] (SPARK-43439) Drop does not work when passed a string with an alias

[ https://issues.apache.org/jira/browse/SPARK-43439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederik Paradis updated SPARK-43439: - Affects Version/s: 3.4.0 (was: 3.3.2) > Drop does not work

[jira] [Created] (SPARK-43439) Drop does not work when passed a string with an alias

Frederik Paradis created SPARK-43439: Summary: Drop does not work when passed a string with an alias Key: SPARK-43439 URL: https://issues.apache.org/jira/browse/SPARK-43439 Project: Spark

[jira] [Updated] (SPARK-43439) Drop does not work when passed a string with an alias

[ https://issues.apache.org/jira/browse/SPARK-43439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederik Paradis updated SPARK-43439: - Description: When passing a string to the drop method, if the string contains an alias,

[jira] [Updated] (SPARK-43439) Drop does not work when passed a string with an alias

[ https://issues.apache.org/jira/browse/SPARK-43439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederik Paradis updated SPARK-43439: - Description: When passing a string to the drop method, if the string contains an alias,

[jira] [Updated] (SPARK-43439) Drop does not work when passed a string with an alias

[ https://issues.apache.org/jira/browse/SPARK-43439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederik Paradis updated SPARK-43439: - Description: When passing a string to the drop method, if the string contains an alias,

[jira] [Created] (SPARK-43438) Fix mismatched column list error on INSERT

Serge Rielau created SPARK-43438: Summary: Fix mismatched column list error on INSERT Key: SPARK-43438 URL: https://issues.apache.org/jira/browse/SPARK-43438 Project: Spark Issue Type:

[jira] [Commented] (SPARK-35198) Add support for calling debugCodegen from Python & Java

[ https://issues.apache.org/jira/browse/SPARK-35198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721475#comment-17721475 ] Ignite TC Bot commented on SPARK-35198: --- User 'juanvisoler' has created a pull request for this

[jira] [Created] (SPARK-43437) Upgrade Arrow to 12.0.0

Yang Jie created SPARK-43437: Summary: Upgrade Arrow to 12.0.0 Key: SPARK-43437 URL: https://issues.apache.org/jira/browse/SPARK-43437 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-43014) Support spark.kubernetes.setSubmitTimeInDriver

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43014: -- Affects Version/s: 3.5.0 (was: 3.4.0)

[jira] [Updated] (SPARK-43014) Support spark.kubernetes.setSubmitTimeInDriver

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43014: -- Summary: Support spark.kubernetes.setSubmitTimeInDriver (was: spark.app.submitTime is not

[jira] [Updated] (SPARK-43014) Support spark.kubernetes.setSubmitTimeInDriver

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43014: -- Priority: Minor (was: Major) > Support spark.kubernetes.setSubmitTimeInDriver >

[jira] [Updated] (SPARK-43014) Support spark.kubernetes.setSubmitTimeInDriver

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43014: -- Issue Type: Improvement (was: Bug) > Support spark.kubernetes.setSubmitTimeInDriver >

[jira] [Resolved] (SPARK-43014) spark.app.submitTime is not right in k8s cluster mode

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43014. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40645

[jira] [Assigned] (SPARK-43014) spark.app.submitTime is not right in k8s cluster mode

[ https://issues.apache.org/jira/browse/SPARK-43014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43014: - Assignee: Zhou Yifan > spark.app.submitTime is not right in k8s cluster mode >

[jira] [Created] (SPARK-43436) Upgrade rocksdbjni to 8.1.1.1

Yang Jie created SPARK-43436: Summary: Upgrade rocksdbjni to 8.1.1.1 Key: SPARK-43436 URL: https://issues.apache.org/jira/browse/SPARK-43436 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-43405) Remove useless code in `ScriptInputOutputSchema`

[ https://issues.apache.org/jira/browse/SPARK-43405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-43405: - Assignee: Jia Fan > Remove useless code in `ScriptInputOutputSchema` >

[jira] [Resolved] (SPARK-43405) Remove useless code in `ScriptInputOutputSchema`

[ https://issues.apache.org/jira/browse/SPARK-43405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-43405. --- Fix Version/s: 3.5.0 Resolution: Fixed > Remove useless code in

[jira] [Assigned] (SPARK-40912) Overhead of Exceptions in DeserializationStream

[ https://issues.apache.org/jira/browse/SPARK-40912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-40912: Assignee: Emil Ejbyfeldt > Overhead of Exceptions in DeserializationStream >

[jira] [Resolved] (SPARK-40912) Overhead of Exceptions in DeserializationStream

[ https://issues.apache.org/jira/browse/SPARK-40912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40912. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 38428

[jira] [Assigned] (SPARK-43434) Disable flaky doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream`

[ https://issues.apache.org/jira/browse/SPARK-43434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43434: Assignee: Ruifeng Zheng > Disable flaky doctest

[jira] [Resolved] (SPARK-43434) Disable flaky doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream`

[ https://issues.apache.org/jira/browse/SPARK-43434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43434. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41114

[jira] [Assigned] (SPARK-37942) Use error classes in the compilation errors of properties

[ https://issues.apache.org/jira/browse/SPARK-37942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-37942: Assignee: Narek Karapetian > Use error classes in the compilation errors of properties >

[jira] [Resolved] (SPARK-37942) Use error classes in the compilation errors of properties

[ https://issues.apache.org/jira/browse/SPARK-37942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37942. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41018

[jira] [Commented] (SPARK-43386) Improve list of suggested column/attributes in `UNRESOLVED_COLUMN.WITH_SUGGESTION` error class

[ https://issues.apache.org/jira/browse/SPARK-43386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721264#comment-17721264 ] ASF GitHub Bot commented on SPARK-43386: User 'vitaliili-db' has created a pull request for this

[jira] [Commented] (SPARK-43386) Improve list of suggested column/attributes in `UNRESOLVED_COLUMN.WITH_SUGGESTION` error class

[ https://issues.apache.org/jira/browse/SPARK-43386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721263#comment-17721263 ] ASF GitHub Bot commented on SPARK-43386: User 'vitaliili-db' has created a pull request for this

[jira] [Assigned] (SPARK-43422) Tags are lost on LogicalRelation when adding _metadata

[ https://issues.apache.org/jira/browse/SPARK-43422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43422: Assignee: Jan-Ole Sasse > Tags are lost on LogicalRelation when adding _metadata >

[jira] [Resolved] (SPARK-43422) Tags are lost on LogicalRelation when adding _metadata

[ https://issues.apache.org/jira/browse/SPARK-43422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43422. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41104

[jira] [Created] (SPARK-43435) re-enable doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream`

Ruifeng Zheng created SPARK-43435: - Summary: re-enable doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream` Key: SPARK-43435 URL: https://issues.apache.org/jira/browse/SPARK-43435 Project:

[jira] [Created] (SPARK-43434) Disable flaky doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream`

Ruifeng Zheng created SPARK-43434: - Summary: Disable flaky doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream` Key: SPARK-43434 URL: https://issues.apache.org/jira/browse/SPARK-43434

[jira] [Updated] (SPARK-43427) Unsigned integer types are deserialized as signed numeric equivalents

[ https://issues.apache.org/jira/browse/SPARK-43427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Upadhyay updated SPARK-43427: --- Description: I'm not sure if "bug" is the correct tag for this jira, but i've tagged it

[jira] [Commented] (SPARK-43357) Spark AWS Glue date partition push down broken

[ https://issues.apache.org/jira/browse/SPARK-43357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721219#comment-17721219 ] Stijn De Haes commented on SPARK-43357: --- Any chance we could backport this to older versions? 3.1,

[jira] [Updated] (SPARK-43425) Add TimestampNTZType to ColumnarBatchRow

[ https://issues.apache.org/jira/browse/SPARK-43425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated SPARK-43425: - Issue Type: Bug (was: Improvement) > Add TimestampNTZType to ColumnarBatchRow >

[jira] [Updated] (SPARK-43400) Add Primary Key syntax support

[ https://issues.apache.org/jira/browse/SPARK-43400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43400: -- Summary: Add Primary Key syntax support (was: create table support the PRIMARY KEY keyword) > Add Primary