[jira] [Assigned] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

[ https://issues.apache.org/jira/browse/SPARK-36184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36184: --- Assignee: Wenchen Fan > Use ValidateRequirements instead of EnsureRequirements to skip AQE

[jira] [Resolved] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

[ https://issues.apache.org/jira/browse/SPARK-36184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36184. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33396

[jira] [Assigned] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36201: Assignee: (was: Apache Spark) > Add check for inner field of schema >

[jira] [Assigned] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36201: Assignee: Apache Spark > Add check for inner field of schema >

[jira] [Commented] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383125#comment-17383125 ] Apache Spark commented on SPARK-36201: -- User 'AngersZh' has created a pull request for this

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas and pandas-on-Spark both have an argument named `mode` in the

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383130#comment-17383130 ] Hyukjin Kwon commented on SPARK-36088: -- You might have to call

[jira] [Commented] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383143#comment-17383143 ] Hyukjin Kwon commented on SPARK-36203: -- Can you show the fullly self-contained reproducer? BTW,

[jira] [Resolved] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36203. -- Resolution: Incomplete > Spark SQL can't use "group by" on the column of map type. >

[jira] [Assigned] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36161: Assignee: Apache Spark > dropDuplicates does not type check argument >

[jira] [Commented] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383175#comment-17383175 ] Apache Spark commented on SPARK-36161: -- User 'sammyjmoseley' has created a pull request for this

[jira] [Assigned] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36161: Assignee: (was: Apache Spark) > dropDuplicates does not type check argument >

[jira] [Updated] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

[ https://issues.apache.org/jira/browse/SPARK-36184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36184: -- Parent: SPARK-33828 Issue Type: Sub-task (was: Improvement) > Use

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Summary: Mapping the `mode` argument to pandas (was: Rename the `mode` argument to avoid

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas on Spark has a argument named `mode` in the APIs below: *

[jira] [Assigned] (SPARK-36197) InputFormat of PartitionDesc is not respected

[ https://issues.apache.org/jira/browse/SPARK-36197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-36197: Assignee: Kent Yao > InputFormat of PartitionDesc is not respected >

[jira] [Resolved] (SPARK-36197) InputFormat of PartitionDesc is not respected

[ https://issues.apache.org/jira/browse/SPARK-36197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-36197. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33406

[jira] [Updated] (SPARK-33844) InsertIntoDir failed since query column name contains ',' cause column type and column names size not equal

[ https://issues.apache.org/jira/browse/SPARK-33844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-33844: -- Parent: SPARK-36200 Issue Type: Sub-task (was: Improvement) > InsertIntoDir failed since

[jira] [Created] (SPARK-36206) Diagnose shuffle data corruption by checksum

wuyi created SPARK-36206: Summary: Diagnose shuffle data corruption by checksum Key: SPARK-36206 URL: https://issues.apache.org/jira/browse/SPARK-36206 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383131#comment-17383131 ] Hyukjin Kwon commented on SPARK-36088: -- cc [~dongjoon] and [~holdenkarau] FYI > 'spark.archives'

[jira] [Commented] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383144#comment-17383144 ] Hyukjin Kwon commented on SPARK-36187: -- For question, let's interact it with Spark mailing list

[jira] [Resolved] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36187. -- Resolution: Incomplete > Commit collision avoidance in dynamicPartitionOverwrite for

[jira] [Reopened] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-35806: - Reopen issue with revised title & description. We should mapping the arguments rather than just

[jira] [Commented] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383046#comment-17383046 ] Haejoon Lee commented on SPARK-35806: - I'm working on this > Mapping the `mode` argument to pandas

[jira] [Commented] (SPARK-36086) The case of the delta table is inconsistent with parquet

[ https://issues.apache.org/jira/browse/SPARK-36086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383195#comment-17383195 ] Wenchen Fan commented on SPARK-36086: - Seems we should improve the v2 describe table command to

[jira] [Assigned] (SPARK-36175) Support TimestampNTZ in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-36175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36175: Assignee: Apache Spark > Support TimestampNTZ in Avro data source >

[jira] [Commented] (SPARK-36175) Support TimestampNTZ in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-36175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383025#comment-17383025 ] Apache Spark commented on SPARK-36175: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36175) Support TimestampNTZ in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-36175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36175: Assignee: (was: Apache Spark) > Support TimestampNTZ in Avro data source >

[jira] [Updated] (SPARK-36206) Diagnose shuffle data corruption by checksum

[ https://issues.apache.org/jira/browse/SPARK-36206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-36206: - Description: After adding checksums in SPARK-35276, we can leverage the checksums to do diagnosis for shuffle

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Summary: Mapping the `mode` argument to pandas in DataFrame.to_csv (was: Mapping the `mode`

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas and pandas-on-Spark both have a argument named `mode` in the

[jira] [Resolved] (SPARK-36134) jackson-databind RCE vulnerability

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36134. -- Resolution: Invalid > jackson-databind RCE vulnerability > --

[jira] [Commented] (SPARK-36185) Implement functions in CategoricalAccessor/CategoricalIndex

[ https://issues.apache.org/jira/browse/SPARK-36185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383146#comment-17383146 ] Hyukjin Kwon commented on SPARK-36185: -- I think it's for Spark 3.2. Most of fixes are being landed

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35806: Assignee: Apache Spark > Mapping the `mode` argument to pandas in DataFrame.to_csv >

[jira] [Commented] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383172#comment-17383172 ] Apache Spark commented on SPARK-35806: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35806: Assignee: (was: Apache Spark) > Mapping the `mode` argument to pandas in

[jira] [Resolved] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34806. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 31905

[jira] [Assigned] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34806: --- Assignee: Enrico Minack > Helper class for batch Dataset.observe() >

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383128#comment-17383128 ] Hyukjin Kwon commented on SPARK-36088: -- does your driver run inside a pod or on a physical host? >

[jira] [Updated] (SPARK-36192) Better error messages when comparing against list

[ https://issues.apache.org/jira/browse/SPARK-36192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36192: - Description: We shall throw TypeError messages rather than Spark exceptions. > Better error

[jira] [Resolved] (SPARK-36163) Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option

[ https://issues.apache.org/jira/browse/SPARK-36163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36163. -- Fix Version/s: 3.3.0 Resolution: Fixed Fixed in

[jira] [Assigned] (SPARK-36163) Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option

[ https://issues.apache.org/jira/browse/SPARK-36163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36163: Assignee: Ivan > Propagate correct JDBC properties in JDBC connector provider and add >

[jira] [Commented] (SPARK-24965) Spark SQL fails when reading a partitioned hive table with different formats per partition

[ https://issues.apache.org/jira/browse/SPARK-24965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383192#comment-17383192 ] tiejiang commented on SPARK-24965: -- I have a similar question, see the link, can anyone answer it,

[jira] [Assigned] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36178: Assignee: Dominik Gehl > Document PySpark Catalog APIs in

[jira] [Resolved] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36178. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33392

[jira] [Assigned] (SPARK-36091) Support TimestampNTZ type in expression TimeWindow

[ https://issues.apache.org/jira/browse/SPARK-36091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-36091: -- Assignee: jiaan.geng > Support TimestampNTZ type in expression TimeWindow >

[jira] [Resolved] (SPARK-36091) Support TimestampNTZ type in expression TimeWindow

[ https://issues.apache.org/jira/browse/SPARK-36091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36091. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33341

[jira] [Updated] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36208: --- Summary: SparkScriptTransformation should support ANSI interval types (was:

[jira] [Commented] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383330#comment-17383330 ] Apache Spark commented on SPARK-36208: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36208: Assignee: Apache Spark (was: Kousuke Saruta) > SparkScriptTransformation should support

[jira] [Commented] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383329#comment-17383329 ] Apache Spark commented on SPARK-36208: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36208: Assignee: Kousuke Saruta (was: Apache Spark) > SparkScriptTransformation should support

[jira] [Created] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

Dominik Gehl created SPARK-36209: Summary: https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc Key: SPARK-36209 URL:

[jira] [Created] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

koert kuipers created SPARK-36210: - Summary: Preserve column insertion order in Dataset.withColumns Key: SPARK-36210 URL: https://issues.apache.org/jira/browse/SPARK-36210 Project: Spark

[jira] [Resolved] (SPARK-36205) Use set-env instead of set-output in GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-36205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36205. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33412

[jira] [Updated] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominik Gehl updated SPARK-36209: - Description: On https://spark.apache.org/docs/latest/sql-programming-guide.html , the link to

[jira] [Assigned] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36209: Assignee: Apache Spark > https://spark.apache.org/docs/latest/sql-programming-guide.html

[jira] [Commented] (SPARK-36166) Support Scala 2.13 test in `dev/run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-36166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383354#comment-17383354 ] Apache Spark commented on SPARK-36166: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36209: Assignee: (was: Apache Spark) >

[jira] [Assigned] (SPARK-36205) Use set-env instead of set-output in GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-36205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36205: Assignee: Hyukjin Kwon > Use set-env instead of set-output in GitHub Actions >

[jira] [Assigned] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36181: Assignee: Dominik Gehl > Update pyspark sql readwriter documentation to Scala level >

[jira] [Resolved] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36181. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33394

[jira] [Resolved] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35806. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33414

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35806: Assignee: Haejoon Lee > Mapping the `mode` argument to pandas in DataFrame.to_csv >

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383322#comment-17383322 ] Apache Spark commented on SPARK-36093: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36207: Assignee: Apache Spark > Export databaseExists in pyspark.sql.catalog >

[jira] [Assigned] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36207: Assignee: (was: Apache Spark) > Export databaseExists in pyspark.sql.catalog >

[jira] [Updated] (SPARK-36208) SparkScriptTransformation

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36208: --- Parent: SPARK-27790 Issue Type: Sub-task (was: Bug) > SparkScriptTransformation >

[jira] [Commented] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383323#comment-17383323 ] Apache Spark commented on SPARK-36207: -- User 'dominikgehl' has created a pull request for this

[jira] [Created] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

Dominik Gehl created SPARK-36207: Summary: Export databaseExists in pyspark.sql.catalog Key: SPARK-36207 URL: https://issues.apache.org/jira/browse/SPARK-36207 Project: Spark Issue Type:

[jira] [Created] (SPARK-36208) SparkScriptTransformation

Kousuke Saruta created SPARK-36208: -- Summary: SparkScriptTransformation Key: SPARK-36208 URL: https://issues.apache.org/jira/browse/SPARK-36208 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383321#comment-17383321 ] Apache Spark commented on SPARK-36093: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383355#comment-17383355 ] Apache Spark commented on SPARK-36209: -- User 'dominikgehl' has created a pull request for this

[jira] [Created] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

Luran He created SPARK-36211: Summary: type check fails for `F.udf(...).asNonDeterministic() Key: SPARK-36211 URL: https://issues.apache.org/jira/browse/SPARK-36211 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luran He updated SPARK-36211: - Description: The following code should type-check, but doesn't: {{import uuid}}

[jira] [Updated] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luran He updated SPARK-36211: - Description: The following code should type-check, but doesn't: {{import uuid}}

[jira] [Commented] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383415#comment-17383415 ] Apache Spark commented on SPARK-36210: -- User 'koertkuipers' has created a pull request for this

[jira] [Assigned] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36210: Assignee: (was: Apache Spark) > Preserve column insertion order in

[jira] [Resolved] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36093. - Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383414#comment-17383414 ] Apache Spark commented on SPARK-36210: -- User 'koertkuipers' has created a pull request for this

[jira] [Assigned] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36210: Assignee: Apache Spark > Preserve column insertion order in Dataset.withColumns >

[jira] [Updated] (SPARK-36212) Add exception for Kafka readstream when decryption fails

[ https://issues.apache.org/jira/browse/SPARK-36212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon LaFlamme updated SPARK-36212: - Fix Version/s: (was: 3.1.0) 3.0.0 > Add exception for Kafka readstream

[jira] [Created] (SPARK-36212) Add exception for Kafka readstream when decryption fails

Jon LaFlamme created SPARK-36212: Summary: Add exception for Kafka readstream when decryption fails Key: SPARK-36212 URL: https://issues.apache.org/jira/browse/SPARK-36212 Project: Spark

[jira] [Assigned] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36211: Assignee: Apache Spark > type check fails for `F.udf(...).asNonDeterministic() >

[jira] [Commented] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383385#comment-17383385 ] Apache Spark commented on SPARK-36211: -- User 'luranhe' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36211: Assignee: (was: Apache Spark) > type check fails for

[jira] [Commented] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383394#comment-17383394 ] Apache Spark commented on SPARK-34806: -- User 'EnricoMi' has created a pull request for this issue:

[jira] [Created] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

Kent Yao created SPARK-36213: Summary: Normalize PartitionSpec for DescTable with PartitionSpec Key: SPARK-36213 URL: https://issues.apache.org/jira/browse/SPARK-36213 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

[ https://issues.apache.org/jira/browse/SPARK-36213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383485#comment-17383485 ] Apache Spark commented on SPARK-36213: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

[ https://issues.apache.org/jira/browse/SPARK-36213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36213: Assignee: Apache Spark > Normalize PartitionSpec for DescTable with PartitionSpec >

[jira] [Assigned] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

[ https://issues.apache.org/jira/browse/SPARK-36213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36213: Assignee: (was: Apache Spark) > Normalize PartitionSpec for DescTable with

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383529#comment-17383529 ] Thomas Graves commented on SPARK-25075: --- Just wanted to check the plans for scala 2.13 in 3.2.  It

[jira] [Resolved] (SPARK-35997) Implement comparison operators for CategoricalDtype in pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-35997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-35997. -- Resolution: Done > Implement comparison operators for CategoricalDtype in pandas API on Spark

[jira] [Commented] (SPARK-36214) Add add_categories to CategoricalAccessor and CategoricalIndex.

[ https://issues.apache.org/jira/browse/SPARK-36214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383641#comment-17383641 ] Takuya Ueshin commented on SPARK-36214: --- I'm working on this. > Add add_categories to

[jira] [Commented] (SPARK-36000) Support creating a ps.Series/Index with `Decimal('NaN')` with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383667#comment-17383667 ] Xinrong Meng commented on SPARK-36000: -- We might want to support

[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383669#comment-17383669 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request for this issue:

[jira] [Resolved] (SPARK-36176) Expose tableExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36176. -- Fix Version/s: 3.2.0 Assignee: Dominik Gehl Resolution: Fixed Fixed in

[jira] [Resolved] (SPARK-36127) Support comparison between a Categorical and a scalar

[ https://issues.apache.org/jira/browse/SPARK-36127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36127. --- Fix Version/s: 3.2.0 Assignee: Xinrong Meng (was: Apache Spark) Resolution:

[jira] [Commented] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383663#comment-17383663 ] Apache Spark commented on SPARK-32919: -- User 'venkata91' has created a pull request for this issue:

[jira] [Commented] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383664#comment-17383664 ] Apache Spark commented on SPARK-32919: -- User 'venkata91' has created a pull request for this issue:

  1   2   >