[jira] [Updated] (SPARK-45145) Add JavaSparkSQLCli example

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45145: --- Labels: pull-request-available (was: ) > Add JavaSparkSQLCli example >

[jira] [Created] (SPARK-45145) Add JavaSparkSQLCli example

2023-09-12 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45145: - Summary: Add JavaSparkSQLCli example Key: SPARK-45145 URL: https://issues.apache.org/jira/browse/SPARK-45145 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45144) Downgrade scala-maven-plugin to 4.7.1

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45144: --- Labels: pull-request-available (was: ) > Downgrade scala-maven-plugin to 4.7.1 >

[jira] [Created] (SPARK-45144) Downgrade scala-maven-plugin to 4.7.1

2023-09-12 Thread Yang Jie (Jira)
Yang Jie created SPARK-45144: Summary: Downgrade scala-maven-plugin to 4.7.1 Key: SPARK-45144 URL: https://issues.apache.org/jira/browse/SPARK-45144 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45065) Support Pandas 2.1.0

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45065: --- Labels: pull-request-available (was: ) > Support Pandas 2.1.0 > > >

[jira] [Updated] (SPARK-45142) Specify the range for Spark Connect dependencies in pyspark base image

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45142: --- Labels: pull-request-available (was: ) > Specify the range for Spark Connect dependencies

[jira] [Created] (SPARK-45143) Make PySpark compatible with PyArrow 13.0.0

2023-09-12 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45143: - Summary: Make PySpark compatible with PyArrow 13.0.0 Key: SPARK-45143 URL: https://issues.apache.org/jira/browse/SPARK-45143 Project: Spark Issue Type:

[jira] [Created] (SPARK-45142) Specify the range for Spark Connect dependencies in pyspark base image

2023-09-12 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-45142: Summary: Specify the range for Spark Connect dependencies in pyspark base image Key: SPARK-45142 URL: https://issues.apache.org/jira/browse/SPARK-45142 Project:

[jira] [Updated] (SPARK-45141) Pin `pyarrow==12.0.1` in CI

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45141: --- Labels: pull-request-available (was: ) > Pin `pyarrow==12.0.1` in CI >

[jira] [Created] (SPARK-45141) Pin `pyarrow==12.0.1` in CI

2023-09-12 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45141: - Summary: Pin `pyarrow==12.0.1` in CI Key: SPARK-45141 URL: https://issues.apache.org/jira/browse/SPARK-45141 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-45140) Support ddl output json format

2023-09-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764488#comment-17764488 ] melin commented on SPARK-45140: --- We have a product that uses jdbc to collect metadata from various

[jira] [Updated] (SPARK-45135) Make utils.eventually a parameterized decorator

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-45135: -- Summary: Make utils.eventually a parameterized decorator (was: Introduce a parameterized

[jira] [Commented] (SPARK-45115) No way to exclude jars setting to classpath while doing spark-submit

2023-09-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764482#comment-17764482 ] Kent Yao commented on SPARK-45115: -- Spark provides a user classpath first feature that might fix your

[jira] [Commented] (SPARK-45140) Support ddl output json format

2023-09-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764468#comment-17764468 ] Kent Yao commented on SPARK-45140: -- Can you detail the feature and usage scenario here? I don't quite

[jira] [Updated] (SPARK-45139) Add DatabricksDialect to handle SQL type conversion

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45139: --- Labels: pull-request-available (was: ) > Add DatabricksDialect to handle SQL type

[jira] [Updated] (SPARK-45140) Support ddl output json format

2023-09-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-45140: -- Description: hive supports ddl output json format. set hive.ddl.output.format=json; hive table metadata is

[jira] [Updated] (SPARK-45140) Support ddl output json format

2023-09-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-45140: -- Description: hive supports ddl output json format. set hive.ddl.output.format=json; hive table metadata is

[jira] [Updated] (SPARK-45140) Support ddl output json format

2023-09-12 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-45140: -- Environment:         was: hive supports ddl output json format. set hive.ddl.output.format=json; hive

[jira] [Assigned] (SPARK-45133) Mark Spark Connect queries as finished when all result tasks are finished, not sent

2023-09-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45133: Assignee: Juliusz Sompolski > Mark Spark Connect queries as finished when all result

[jira] [Created] (SPARK-45140) Support ddl output json format

2023-09-12 Thread melin (Jira)
melin created SPARK-45140: - Summary: Support ddl output json format Key: SPARK-45140 URL: https://issues.apache.org/jira/browse/SPARK-45140 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-45133) Mark Spark Connect queries as finished when all result tasks are finished, not sent

2023-09-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45133. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42889

[jira] [Created] (SPARK-45139) Add DatabricksDialect to handle SQL type conversion

2023-09-12 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-45139: Summary: Add DatabricksDialect to handle SQL type conversion Key: SPARK-45139 URL: https://issues.apache.org/jira/browse/SPARK-45139 Project: Spark Issue

[jira] [Updated] (SPARK-45139) Add DatabricksDialect to handle SQL type conversion

2023-09-12 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-45139: - Description: Databricks SQL dialect is needed to refine type conversion when connecting to a

[jira] [Comment Edited] (SPARK-44580) RocksDB crashed when testing in GitHub Actions

2023-09-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764450#comment-17764450 ] BingKun Pan edited comment on SPARK-44580 at 9/13/23 2:21 AM: --

[jira] [Updated] (SPARK-45096) Optimize apt-get install in Dockerfile

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45096: --- Labels: pull-request-available (was: ) > Optimize apt-get install in Dockerfile >

[jira] [Assigned] (SPARK-45096) Optimize apt-get install in Dockerfile

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45096: - Assignee: Ruifeng Zheng > Optimize apt-get install in Dockerfile >

[jira] [Resolved] (SPARK-45096) Optimize apt-get install in Dockerfile

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45096. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42842

[jira] [Resolved] (SPARK-45131) Refine docstring of `ceil/ceiling/floor/round/bround`

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45131. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42892

[jira] [Assigned] (SPARK-45131) Refine docstring of `ceil/ceiling/floor/round/bround`

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45131: - Assignee: BingKun Pan > Refine docstring of `ceil/ceiling/floor/round/bround` >

[jira] [Commented] (SPARK-44580) RocksDB crashed when testing in GitHub Actions

2023-09-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764450#comment-17764450 ] BingKun Pan commented on SPARK-44580: - [https://github.com/apache/spark/pull/42862] All errors

[jira] [Resolved] (SPARK-45120) Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI

2023-09-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-45120. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42879

[jira] [Resolved] (SPARK-41281) Feature parity: SparkSession API in Spark Connect

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41281. --- Resolution: Resolved > Feature parity: SparkSession API in Spark Connect >

[jira] [Updated] (SPARK-41811) Implement SparkSession.sql's string formatter

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41811: -- Affects Version/s: 4.0.0 (was: 3.4.0) > Implement

[jira] [Updated] (SPARK-41811) Implement SparkSession.sql's string formatter

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41811: -- Epic Link: SPARK-39375 > Implement SparkSession.sql's string formatter >

[jira] [Updated] (SPARK-41811) Implement SparkSession.sql's string formatter

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41811: -- Parent: (was: SPARK-41281) Issue Type: New Feature (was: Sub-task) > Implement

[jira] [Commented] (SPARK-41842) Support data type Timestamp(NANOSECOND, null)

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764441#comment-17764441 ] Ruifeng Zheng commented on SPARK-41842: --- resolved since we had enable ^^^ tests > Support data

[jira] [Resolved] (SPARK-41842) Support data type Timestamp(NANOSECOND, null)

2023-09-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41842. --- Fix Version/s: 3.4.0 Resolution: Resolved > Support data type Timestamp(NANOSECOND,

[jira] [Updated] (SPARK-45123) Raise TypeError for DataFrame.interpolate when all columns are object-dtype.

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45123: --- Labels: pull-request-available (was: ) > Raise TypeError for DataFrame.interpolate when

[jira] [Updated] (SPARK-42439) Job description in v2 FileWrites can have the wrong committer

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-42439: --- Labels: bug pull-request-available (was: bug) > Job description in v2 FileWrites can have

[jira] [Updated] (SPARK-45138) Define a new error class and apply for the case where checkpointing state to DFS fails

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45138: --- Labels: pull-request-available (was: ) > Define a new error class and apply for the case

[jira] [Created] (SPARK-45138) Define a new error class and apply for the case where checkpointing state to DFS fails

2023-09-12 Thread Raghu Angadi (Jira)
Raghu Angadi created SPARK-45138: Summary: Define a new error class and apply for the case where checkpointing state to DFS fails Key: SPARK-45138 URL: https://issues.apache.org/jira/browse/SPARK-45138

[jira] [Updated] (SPARK-40559) Add applyInArrow to pyspark.sql.GroupedData

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-40559: --- Labels: pull-request-available (was: ) > Add applyInArrow to pyspark.sql.GroupedData >

[jira] [Updated] (SPARK-44459) Garbage collection doesn't include finalization run

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44459: --- Labels: MemoryLeak gzip memory-bug memory-control pull-request-available

[jira] [Updated] (SPARK-44872) Testing reattachable execute

2023-09-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-44872: -- Fix Version/s: 3.5.1 > Testing reattachable execute > >

[jira] [Commented] (SPARK-42890) Add Identifier to the InMemoryTableScan node on the SQL page

2023-09-12 Thread Hudson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764290#comment-17764290 ] Hudson commented on SPARK-42890: User 'yliou' has created a pull request for this issue:

[jira] [Commented] (SPARK-43152) User-defined output metadata path (_spark_metadata)

2023-09-12 Thread Hudson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764291#comment-17764291 ] Hudson commented on SPARK-43152: User 'woj-i' has created a pull request for this issue:

[jira] [Created] (SPARK-45137) Unsupported map and array constructors by `sql()` in connect clients

2023-09-12 Thread Max Gekk (Jira)
Max Gekk created SPARK-45137: Summary: Unsupported map and array constructors by `sql()` in connect clients Key: SPARK-45137 URL: https://issues.apache.org/jira/browse/SPARK-45137 Project: Spark

[jira] [Resolved] (SPARK-45117) Implement missing otherCopyArgs for the MultiCommutativeOp expression

2023-09-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45117. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-45117) Implement missing otherCopyArgs for the MultiCommutativeOp expression

2023-09-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45117: --- Assignee: Supun Nakandala > Implement missing otherCopyArgs for the MultiCommutativeOp

[jira] [Updated] (SPARK-45110) Upgrade rocksdbjni to 8.5.3

2023-09-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45110: -- Fix Version/s: (was: 4.0.0) (was: 3.5.1) > Upgrade rocksdbjni to

[jira] [Commented] (SPARK-45110) Upgrade rocksdbjni to 8.5.3

2023-09-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764259#comment-17764259 ] Dongjoon Hyun commented on SPARK-45110: --- This is reverted due to the following.

[jira] [Reopened] (SPARK-45110) Upgrade rocksdbjni to 8.5.3

2023-09-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-45110: --- Assignee: (was: BingKun Pan) > Upgrade rocksdbjni to 8.5.3 >

[jira] [Updated] (SPARK-44910) Encoders.bean does not support superclasses with generic type arguments

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44910: --- Labels: pull-request-available (was: ) > Encoders.bean does not support superclasses with

[jira] [Resolved] (SPARK-44872) Testing reattachable execute

2023-09-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44872. --- Fix Version/s: 4.0.0 Assignee: Juliusz Sompolski Resolution: Fixed

[jira] [Resolved] (SPARK-45081) Encoders.bean does no longer work with read-only properties

2023-09-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-45081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-45081. --- Fix Version/s: 4.0.0 3.5.1 Resolution: Fixed >

[jira] [Updated] (SPARK-45081) Encoders.bean does no longer work with read-only properties

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45081: --- Labels: pull-request-available (was: ) > Encoders.bean does no longer work with read-only

[jira] [Created] (SPARK-45136) Improve ClosureCleaner to support closures defined in Ammonite REPL

2023-09-12 Thread Vsevolod Stepanov (Jira)
Vsevolod Stepanov created SPARK-45136: - Summary: Improve ClosureCleaner to support closures defined in Ammonite REPL Key: SPARK-45136 URL: https://issues.apache.org/jira/browse/SPARK-45136

[jira] [Commented] (SPARK-45115) No way to exclude jars setting to classpath while doing spark-submit

2023-09-12 Thread Sumanto Pal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764198#comment-17764198 ] Sumanto Pal commented on SPARK-45115: - Challenge is we are using Firm recommended FAT jar which has

[jira] [Comment Edited] (SPARK-45115) No way to exclude jars setting to classpath while doing spark-submit

2023-09-12 Thread Sumanto Pal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764198#comment-17764198 ] Sumanto Pal edited comment on SPARK-45115 at 9/12/23 1:55 PM: -- Challenge is

[jira] [Updated] (SPARK-45131) Refine docstring of `ceil/ceiling/floor/round/bround`

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45131: --- Labels: pull-request-available (was: ) > Refine docstring of

[jira] [Updated] (SPARK-45135) Introduce a parameterized decorator for test retry

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45135: --- Labels: pull-request-available (was: ) > Introduce a parameterized decorator for test

[jira] [Created] (SPARK-45135) Introduce a parameterized decorator for test retry

2023-09-12 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45135: - Summary: Introduce a parameterized decorator for test retry Key: SPARK-45135 URL: https://issues.apache.org/jira/browse/SPARK-45135 Project: Spark Issue

[jira] [Updated] (SPARK-45055) Do not transpose windows if they conflict on ORDER BY / PROJECT clauses

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45055: --- Labels: pull-request-available (was: ) > Do not transpose windows if they conflict on

[jira] [Updated] (SPARK-25689) Move token renewal logic to driver in yarn-client mode

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-25689: --- Labels: pull-request-available (was: ) > Move token renewal logic to driver in yarn-client

[jira] [Commented] (SPARK-45134) Data duplication may occur when fallback to origin shuffle block

2023-09-12 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764130#comment-17764130 ] gaoyajun02 commented on SPARK-45134: Hi, [~csingh] [~vsowrirajan] [~mshen] , Can you take a look?

[jira] [Updated] (SPARK-45134) Data duplication may occur when fallback to origin shuffle block

2023-09-12 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-45134: --- Description: One possible situation that has been found is that, during the process of requesting

[jira] [Updated] (SPARK-45134) Data duplication may occur when fallback to origin shuffle block

2023-09-12 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-45134: --- Description: One possible situation that has been found is that, during the process of requesting

[jira] [Updated] (SPARK-45134) Data duplication may occur when fallback to origin shuffle block

2023-09-12 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-45134: --- Description: One possible situation that has been found is that, during the process of requesting

[jira] [Created] (SPARK-45134) Data duplication may occur when fallback to origin shuffle block

2023-09-12 Thread gaoyajun02 (Jira)
gaoyajun02 created SPARK-45134: -- Summary: Data duplication may occur when fallback to origin shuffle block Key: SPARK-45134 URL: https://issues.apache.org/jira/browse/SPARK-45134 Project: Spark

[jira] [Updated] (SPARK-44872) Testing reattachable execute

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44872: --- Labels: pull-request-available (was: ) > Testing reattachable execute >

[jira] [Updated] (SPARK-45133) Mark Spark Connect queries as finished when all result tasks are finished, not sent

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45133: --- Labels: pull-request-available (was: ) > Mark Spark Connect queries as finished when all

[jira] [Updated] (SPARK-44776) Add ProducedRowCount to SparkListenerConnectOperationFinished

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44776: --- Labels: pull-request-available (was: ) > Add ProducedRowCount to

[jira] [Created] (SPARK-45133) Mark Spark Connect queries as finished when all result tasks are finished, not sent

2023-09-12 Thread Juliusz Sompolski (Jira)
Juliusz Sompolski created SPARK-45133: - Summary: Mark Spark Connect queries as finished when all result tasks are finished, not sent Key: SPARK-45133 URL: https://issues.apache.org/jira/browse/SPARK-45133

[jira] [Commented] (SPARK-45115) No way to exclude jars setting to classpath while doing spark-submit

2023-09-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764115#comment-17764115 ] Kent Yao commented on SPARK-45115: -- Build your spark app with slf4j provided scope? > No way to

[jira] [Updated] (SPARK-45132) Fix IDENTIFIER clause for functions

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45132: --- Labels: pull-request-available (was: ) > Fix IDENTIFIER clause for functions >

[jira] [Created] (SPARK-45132) Fix IDENTIFIER clause for functions

2023-09-12 Thread Serge Rielau (Jira)
Serge Rielau created SPARK-45132: Summary: Fix IDENTIFIER clause for functions Key: SPARK-45132 URL: https://issues.apache.org/jira/browse/SPARK-45132 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-45034) Support deterministic mode function

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45034: -- Assignee: (was: Apache Spark) > Support deterministic mode function >

[jira] [Assigned] (SPARK-45034) Support deterministic mode function

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45034: -- Assignee: Apache Spark > Support deterministic mode function >

[jira] [Assigned] (SPARK-45034) Support deterministic mode function

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45034: -- Assignee: (was: Apache Spark) > Support deterministic mode function >

[jira] [Assigned] (SPARK-45034) Support deterministic mode function

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45034: -- Assignee: Apache Spark > Support deterministic mode function >

[jira] [Created] (SPARK-45131) Refine docstring of `ceil/ceiling/floor/round/bround`

2023-09-12 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-45131: --- Summary: Refine docstring of `ceil/ceiling/floor/round/bround` Key: SPARK-45131 URL: https://issues.apache.org/jira/browse/SPARK-45131 Project: Spark Issue

[jira] [Assigned] (SPARK-45112) Use UnresolvedFunction based resolution in SQL Dataset functions

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45112: -- Assignee: (was: Apache Spark) > Use UnresolvedFunction based resolution in SQL

[jira] [Assigned] (SPARK-45112) Use UnresolvedFunction based resolution in SQL Dataset functions

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45112: -- Assignee: Apache Spark > Use UnresolvedFunction based resolution in SQL Dataset

[jira] [Updated] (SPARK-45130) Avoid Spark connect ML model to change input pandas dataframe

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45130: --- Labels: pull-request-available (was: ) > Avoid Spark connect ML model to change input

[jira] [Resolved] (SPARK-44911) create hive table with invalid column should return error class

2023-09-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44911. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42609

[jira] [Assigned] (SPARK-44911) create hive table with invalid column should return error class

2023-09-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-44911: Assignee: ming95 > create hive table with invalid column should return error class >

[jira] [Updated] (SPARK-44911) create hive table with invalid column should return error class

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44911: --- Labels: pull-request-available (was: ) > create hive table with invalid column should

[jira] [Updated] (SPARK-38215) InsertIntoHiveDir support convert metadata

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-38215: --- Labels: pull-request-available (was: ) > InsertIntoHiveDir support convert metadata >

[jira] [Assigned] (SPARK-45092) Avoid analyze twice for failed queries

2023-09-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-45092: Assignee: Kent Yao > Avoid analyze twice for failed queries >

[jira] [Resolved] (SPARK-45092) Avoid analyze twice for failed queries

2023-09-12 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-45092. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42838

[jira] [Updated] (SPARK-44622) Implement error enrichment and JVM stacktrace

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44622: --- Labels: pull-request-available (was: ) > Implement error enrichment and JVM stacktrace >

[jira] [Created] (SPARK-45130) Avoid Spark connect ML model to change input pandas dataframe

2023-09-12 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-45130: -- Summary: Avoid Spark connect ML model to change input pandas dataframe Key: SPARK-45130 URL: https://issues.apache.org/jira/browse/SPARK-45130 Project: Spark

[jira] [Updated] (SPARK-45130) Avoid Spark connect ML model to change input pandas dataframe

2023-09-12 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-45130: --- Description: Currently,  > Avoid Spark connect ML model to change input pandas dataframe >

[jira] [Updated] (SPARK-45129) Add pyspark "ml-connect" extras dependencies

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45129: --- Labels: pull-request-available (was: ) > Add pyspark "ml-connect" extras dependencies >

[jira] [Created] (SPARK-45129) Add pyspark "ml-connect" extras dependencies

2023-09-12 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-45129: -- Summary: Add pyspark "ml-connect" extras dependencies Key: SPARK-45129 URL: https://issues.apache.org/jira/browse/SPARK-45129 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45034) Support deterministic mode function

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45034: --- Labels: pull-request-available (was: ) > Support deterministic mode function >

[jira] [Resolved] (SPARK-44915) Validate checksum of remounted PVC's shuffle data before recovery

2023-09-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44915. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42724

[jira] [Assigned] (SPARK-44915) Validate checksum of remounted PVC's shuffle data before recovery

2023-09-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44915: - Assignee: Dongjoon Hyun > Validate checksum of remounted PVC's shuffle data before

[jira] [Updated] (SPARK-45128) Support CalendarIntervalType in Arrow

2023-09-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45128: --- Labels: pull-request-available (was: ) > Support CalendarIntervalType in Arrow >

[jira] [Resolved] (SPARK-45085) Merge UNSUPPORTED_TEMP_VIEW_OPERATION into UNSUPPORTED_VIEW_OPERATION and refactor some logic

2023-09-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-45085. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42824

  1   2   >