[jira] [Assigned] (SPARK-36197) InputFormat of PartitionDesc is not respected

[ https://issues.apache.org/jira/browse/SPARK-36197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-36197: Assignee: Kent Yao > InputFormat of PartitionDesc is not respected >

[jira] [Resolved] (SPARK-36197) InputFormat of PartitionDesc is not respected

[ https://issues.apache.org/jira/browse/SPARK-36197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-36197. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33406 [https://github.com

[jira] [Assigned] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36201: Assignee: (was: Apache Spark) > Add check for inner field of schema > ---

[jira] [Assigned] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36201: Assignee: Apache Spark > Add check for inner field of schema > --

[jira] [Commented] (SPARK-36201) Add check for inner field of schema

[ https://issues.apache.org/jira/browse/SPARK-36201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383125#comment-17383125 ] Apache Spark commented on SPARK-36201: -- User 'AngersZh' has created a pull requ

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas and pandas-on-Spark both have a argument named `mode` in the [DataFrame.to_cs

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Summary: Mapping the `mode` argument to pandas in DataFrame.to_csv (was: Mapping the `mode` argum

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383128#comment-17383128 ] Hyukjin Kwon commented on SPARK-36088: -- does your driver run inside a pod or on a p

[jira] [Updated] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35806: Description: pandas and pandas-on-Spark both have an argument named `mode` in the [DataFrame.to_c

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383130#comment-17383130 ] Hyukjin Kwon commented on SPARK-36088: -- You might have to call https://github.com/

[jira] [Commented] (SPARK-36088) 'spark.archives' does not extract the archive file into the driver under client mode

[ https://issues.apache.org/jira/browse/SPARK-36088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383131#comment-17383131 ] Hyukjin Kwon commented on SPARK-36088: -- cc [~dongjoon] and [~holdenkarau] FYI > 's

[jira] [Resolved] (SPARK-36134) jackson-databind RCE vulnerability

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36134. -- Resolution: Invalid > jackson-databind RCE vulnerability > --

[jira] [Commented] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383143#comment-17383143 ] Hyukjin Kwon commented on SPARK-36203: -- Can you show the fullly self-contained repr

[jira] [Resolved] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36203. -- Resolution: Incomplete > Spark SQL can't use "group by" on the column of map type. > -

[jira] [Updated] (SPARK-36192) Better error messages when comparing against list

[ https://issues.apache.org/jira/browse/SPARK-36192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36192: - Description: We shall throw TypeError messages rather than Spark exceptions. > Better error mess

[jira] [Commented] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383144#comment-17383144 ] Hyukjin Kwon commented on SPARK-36187: -- For question, let's interact it with Spark

[jira] [Resolved] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36187. -- Resolution: Incomplete > Commit collision avoidance in dynamicPartitionOverwrite for non-Parqu

[jira] [Commented] (SPARK-36185) Implement functions in CategoricalAccessor/CategoricalIndex

[ https://issues.apache.org/jira/browse/SPARK-36185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383146#comment-17383146 ] Hyukjin Kwon commented on SPARK-36185: -- I think it's for Spark 3.2. Most of fixes a

[jira] [Resolved] (SPARK-36163) Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option

[ https://issues.apache.org/jira/browse/SPARK-36163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36163. -- Fix Version/s: 3.3.0 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/33

[jira] [Assigned] (SPARK-36163) Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option

[ https://issues.apache.org/jira/browse/SPARK-36163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36163: Assignee: Ivan > Propagate correct JDBC properties in JDBC connector provider and add >

[jira] [Commented] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383172#comment-17383172 ] Apache Spark commented on SPARK-35806: -- User 'itholic' has created a pull request f

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35806: Assignee: (was: Apache Spark) > Mapping the `mode` argument to pandas in DataFrame.to

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35806: Assignee: Apache Spark > Mapping the `mode` argument to pandas in DataFrame.to_csv >

[jira] [Assigned] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36161: Assignee: Apache Spark > dropDuplicates does not type check argument > --

[jira] [Commented] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383175#comment-17383175 ] Apache Spark commented on SPARK-36161: -- User 'sammyjmoseley' has created a pull req

[jira] [Assigned] (SPARK-36161) dropDuplicates does not type check argument

[ https://issues.apache.org/jira/browse/SPARK-36161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36161: Assignee: (was: Apache Spark) > dropDuplicates does not type check argument > ---

[jira] [Commented] (SPARK-24965) Spark SQL fails when reading a partitioned hive table with different formats per partition

[ https://issues.apache.org/jira/browse/SPARK-24965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383192#comment-17383192 ] tiejiang commented on SPARK-24965: -- I have a similar question, see the link, can anyone

[jira] [Resolved] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34806. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 31905 [https://gith

[jira] [Assigned] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34806: --- Assignee: Enrico Minack > Helper class for batch Dataset.observe() > --

[jira] [Commented] (SPARK-36086) The case of the delta table is inconsistent with parquet

[ https://issues.apache.org/jira/browse/SPARK-36086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383195#comment-17383195 ] Wenchen Fan commented on SPARK-36086: - Seems we should improve the v2 describe table

[jira] [Assigned] (SPARK-36205) Use set-env instead of set-output in GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-36205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36205: Assignee: Hyukjin Kwon > Use set-env instead of set-output in GitHub Actions > --

[jira] [Assigned] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36178: Assignee: Dominik Gehl > Document PySpark Catalog APIs in docs/source/reference/pyspark.s

[jira] [Resolved] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36178. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33392 [https://gi

[jira] [Assigned] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36181: Assignee: Dominik Gehl > Update pyspark sql readwriter documentation to Scala level > ---

[jira] [Resolved] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36181. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33394 [https://gi

[jira] [Resolved] (SPARK-36205) Use set-env instead of set-output in GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-36205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36205. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33412 [https://gi

[jira] [Resolved] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35806. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33414 [https://gi

[jira] [Assigned] (SPARK-35806) Mapping the `mode` argument to pandas in DataFrame.to_csv

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35806: Assignee: Haejoon Lee > Mapping the `mode` argument to pandas in DataFrame.to_csv > -

[jira] [Assigned] (SPARK-36091) Support TimestampNTZ type in expression TimeWindow

[ https://issues.apache.org/jira/browse/SPARK-36091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-36091: -- Assignee: jiaan.geng > Support TimestampNTZ type in expression TimeWindow >

[jira] [Resolved] (SPARK-36091) Support TimestampNTZ type in expression TimeWindow

[ https://issues.apache.org/jira/browse/SPARK-36091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36091. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33341 [https:

[jira] [Created] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

Dominik Gehl created SPARK-36207: Summary: Export databaseExists in pyspark.sql.catalog Key: SPARK-36207 URL: https://issues.apache.org/jira/browse/SPARK-36207 Project: Spark Issue Type: Impr

[jira] [Created] (SPARK-36208) SparkScriptTransformation

Kousuke Saruta created SPARK-36208: -- Summary: SparkScriptTransformation Key: SPARK-36208 URL: https://issues.apache.org/jira/browse/SPARK-36208 Project: Spark Issue Type: Bug Comp

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383321#comment-17383321 ] Apache Spark commented on SPARK-36093: -- User 'AngersZh' has created a pull requ

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383322#comment-17383322 ] Apache Spark commented on SPARK-36093: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36207: Assignee: Apache Spark > Export databaseExists in pyspark.sql.catalog > -

[jira] [Assigned] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36207: Assignee: (was: Apache Spark) > Export databaseExists in pyspark.sql.catalog > --

[jira] [Updated] (SPARK-36208) SparkScriptTransformation

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36208: --- Parent: SPARK-27790 Issue Type: Sub-task (was: Bug) > SparkScriptTransformation >

[jira] [Commented] (SPARK-36207) Export databaseExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383323#comment-17383323 ] Apache Spark commented on SPARK-36207: -- User 'dominikgehl' has created a pull reque

[jira] [Updated] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36208: --- Summary: SparkScriptTransformation should support ANSI interval types (was: SparkScriptTran

[jira] [Assigned] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36208: Assignee: Apache Spark (was: Kousuke Saruta) > SparkScriptTransformation should support

[jira] [Commented] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383329#comment-17383329 ] Apache Spark commented on SPARK-36208: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36208: Assignee: Kousuke Saruta (was: Apache Spark) > SparkScriptTransformation should support

[jira] [Commented] (SPARK-36208) SparkScriptTransformation should support ANSI interval types

[ https://issues.apache.org/jira/browse/SPARK-36208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383330#comment-17383330 ] Apache Spark commented on SPARK-36208: -- User 'sarutak' has created a pull request f

[jira] [Created] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

Dominik Gehl created SPARK-36209: Summary: https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc Key: SPARK-36209 URL: https://issues.apache.org/jira/browse/SPARK-362

[jira] [Updated] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominik Gehl updated SPARK-36209: - Description: On https://spark.apache.org/docs/latest/sql-programming-guide.html , the link to t

[jira] [Commented] (SPARK-36166) Support Scala 2.13 test in `dev/run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-36166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383354#comment-17383354 ] Apache Spark commented on SPARK-36166: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36209: Assignee: (was: Apache Spark) > https://spark.apache.org/docs/latest/sql-programming-

[jira] [Assigned] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36209: Assignee: Apache Spark > https://spark.apache.org/docs/latest/sql-programming-guide.html

[jira] [Commented] (SPARK-36209) https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc

[ https://issues.apache.org/jira/browse/SPARK-36209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383355#comment-17383355 ] Apache Spark commented on SPARK-36209: -- User 'dominikgehl' has created a pull reque

[jira] [Created] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

koert kuipers created SPARK-36210: - Summary: Preserve column insertion order in Dataset.withColumns Key: SPARK-36210 URL: https://issues.apache.org/jira/browse/SPARK-36210 Project: Spark Issu

[jira] [Updated] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luran He updated SPARK-36211: - Description: The following code should type-check, but doesn't: {{import uuid}} {{pyspark.sql.function

[jira] [Updated] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luran He updated SPARK-36211: - Description: The following code should type-check, but doesn't: {{import uuid}} {{pyspark.sql.fun

[jira] [Created] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

Luran He created SPARK-36211: Summary: type check fails for `F.udf(...).asNonDeterministic() Key: SPARK-36211 URL: https://issues.apache.org/jira/browse/SPARK-36211 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36211: Assignee: Apache Spark > type check fails for `F.udf(...).asNonDeterministic() >

[jira] [Commented] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383385#comment-17383385 ] Apache Spark commented on SPARK-36211: -- User 'luranhe' has created a pull request f

[jira] [Assigned] (SPARK-36211) type check fails for `F.udf(...).asNonDeterministic()

[ https://issues.apache.org/jira/browse/SPARK-36211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36211: Assignee: (was: Apache Spark) > type check fails for `F.udf(...).asNonDeterministic()

[jira] [Commented] (SPARK-34806) Helper class for batch Dataset.observe()

[ https://issues.apache.org/jira/browse/SPARK-34806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383394#comment-17383394 ] Apache Spark commented on SPARK-34806: -- User 'EnricoMi' has created a pull request

[jira] [Resolved] (SPARK-36093) The result incorrect if the partition path case is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36093. - Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed Issue resolved by pull re

[jira] [Commented] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383414#comment-17383414 ] Apache Spark commented on SPARK-36210: -- User 'koertkuipers' has created a pull requ

[jira] [Assigned] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36210: Assignee: Apache Spark > Preserve column insertion order in Dataset.withColumns > ---

[jira] [Assigned] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36210: Assignee: (was: Apache Spark) > Preserve column insertion order in Dataset.withColumn

[jira] [Commented] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

[ https://issues.apache.org/jira/browse/SPARK-36210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383415#comment-17383415 ] Apache Spark commented on SPARK-36210: -- User 'koertkuipers' has created a pull requ

[jira] [Created] (SPARK-36212) Add exception for Kafka readstream when decryption fails

Jon LaFlamme created SPARK-36212: Summary: Add exception for Kafka readstream when decryption fails Key: SPARK-36212 URL: https://issues.apache.org/jira/browse/SPARK-36212 Project: Spark Issu

[jira] [Updated] (SPARK-36212) Add exception for Kafka readstream when decryption fails

[ https://issues.apache.org/jira/browse/SPARK-36212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon LaFlamme updated SPARK-36212: - Fix Version/s: (was: 3.1.0) 3.0.0 > Add exception for Kafka readstream wh

[jira] [Created] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

Kent Yao created SPARK-36213: Summary: Normalize PartitionSpec for DescTable with PartitionSpec Key: SPARK-36213 URL: https://issues.apache.org/jira/browse/SPARK-36213 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

[ https://issues.apache.org/jira/browse/SPARK-36213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383485#comment-17383485 ] Apache Spark commented on SPARK-36213: -- User 'yaooqinn' has created a pull request

[jira] [Assigned] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

[ https://issues.apache.org/jira/browse/SPARK-36213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36213: Assignee: Apache Spark > Normalize PartitionSpec for DescTable with PartitionSpec > -

[jira] [Assigned] (SPARK-36213) Normalize PartitionSpec for DescTable with PartitionSpec

[ https://issues.apache.org/jira/browse/SPARK-36213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36213: Assignee: (was: Apache Spark) > Normalize PartitionSpec for DescTable with PartitionS

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383529#comment-17383529 ] Thomas Graves commented on SPARK-25075: --- Just wanted to check the plans for scala

[jira] [Resolved] (SPARK-36127) Support comparison between a Categorical and a scalar

[ https://issues.apache.org/jira/browse/SPARK-36127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36127. --- Fix Version/s: 3.2.0 Assignee: Xinrong Meng (was: Apache Spark) Resolution:

[jira] [Resolved] (SPARK-35997) Implement comparison operators for CategoricalDtype in pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-35997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-35997. -- Resolution: Done > Implement comparison operators for CategoricalDtype in pandas API on Spark

[jira] [Created] (SPARK-36214) Add add_categories to CategoricalAccessor and CategoricalIndex.

Takuya Ueshin created SPARK-36214: - Summary: Add add_categories to CategoricalAccessor and CategoricalIndex. Key: SPARK-36214 URL: https://issues.apache.org/jira/browse/SPARK-36214 Project: Spark

[jira] [Commented] (SPARK-36214) Add add_categories to CategoricalAccessor and CategoricalIndex.

[ https://issues.apache.org/jira/browse/SPARK-36214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383641#comment-17383641 ] Takuya Ueshin commented on SPARK-36214: --- I'm working on this. > Add add_categorie

[jira] [Commented] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383663#comment-17383663 ] Apache Spark commented on SPARK-32919: -- User 'venkata91' has created a pull request

[jira] [Commented] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383664#comment-17383664 ] Apache Spark commented on SPARK-32919: -- User 'venkata91' has created a pull request

[jira] [Commented] (SPARK-36000) Support creating a ps.Series/Index with `Decimal('NaN')` with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383667#comment-17383667 ] Xinrong Meng commented on SPARK-36000: -- We might want to support spark.createDataFr

[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383669#comment-17383669 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request

[jira] [Resolved] (SPARK-36176) Expose tableExists in pyspark.sql.catalog

[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36176. -- Fix Version/s: 3.2.0 Assignee: Dominik Gehl Resolution: Fixed Fixed in https:/

[jira] [Commented] (SPARK-35809) Add `index_col` argument for ps.sql.

[ https://issues.apache.org/jira/browse/SPARK-35809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383679#comment-17383679 ] Haejoon Lee commented on SPARK-35809: - I'm working on this > Add `index_col` argume

[jira] [Assigned] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

[ https://issues.apache.org/jira/browse/SPARK-36179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36179: Assignee: Kent Yao > Support TimestampNTZType in SparkGetColumnsOperation > -

[jira] [Resolved] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

[ https://issues.apache.org/jira/browse/SPARK-36179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36179. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33393 [https://gi

[jira] [Updated] (SPARK-35809) Add `index_col` argument for ps.sql.

[ https://issues.apache.org/jira/browse/SPARK-35809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-35809: Description: The current behavior of [ps.sql  |https://koalas.readthedocs.io/en/latest/reference/ap

[jira] [Created] (SPARK-36215) Add logging for slow fetches to diagnose external shuffle service issues

Shardul Mahadik created SPARK-36215: --- Summary: Add logging for slow fetches to diagnose external shuffle service issues Key: SPARK-36215 URL: https://issues.apache.org/jira/browse/SPARK-36215 Projec

[jira] [Commented] (SPARK-35807) Deprecate the `num_files` argument

[ https://issues.apache.org/jira/browse/SPARK-35807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383698#comment-17383698 ] Apache Spark commented on SPARK-35807: -- User 'itholic' has created a pull request f

[jira] [Commented] (SPARK-35807) Deprecate the `num_files` argument

[ https://issues.apache.org/jira/browse/SPARK-35807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383699#comment-17383699 ] Apache Spark commented on SPARK-35807: -- User 'itholic' has created a pull request f

[jira] [Updated] (SPARK-36216) Increase timeout for StreamingLinearRegressionWithTests.test_parameter_convergence

[ https://issues.apache.org/jira/browse/SPARK-36216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36216: - Docs Text: (was: Test is flaky (https://github.com/apache/spark/runs/3109815586): {code} Trac

[jira] [Updated] (SPARK-36216) Increase timeout for StreamingLinearRegressionWithTests.test_parameter_convergence

[ https://issues.apache.org/jira/browse/SPARK-36216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36216: - Description: Test is flaky (https://github.com/apache/spark/runs/3109815586): {code} Traceback

[jira] [Created] (SPARK-36216) Increase timeout for StreamingLinearRegressionWithTests.test_parameter_convergence

Hyukjin Kwon created SPARK-36216: Summary: Increase timeout for StreamingLinearRegressionWithTests.test_parameter_convergence Key: SPARK-36216 URL: https://issues.apache.org/jira/browse/SPARK-36216 Pr

[jira] [Created] (SPARK-36217) Rename CustomShuffleReader and OptimizeLocalShuffleReader

Hyukjin Kwon created SPARK-36217: Summary: Rename CustomShuffleReader and OptimizeLocalShuffleReader Key: SPARK-36217 URL: https://issues.apache.org/jira/browse/SPARK-36217 Project: Spark Iss

[jira] [Assigned] (SPARK-36216) Increase timeout for StreamingLinearRegressionWithTests.test_parameter_convergence

[ https://issues.apache.org/jira/browse/SPARK-36216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36216: Assignee: (was: Apache Spark) > Increase timeout for > StreamingLinearRegressionWith

  1   2   >