[jira] [Commented] (SPARK-36713) Document new syntax for specifying index type

[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425347#comment-17425347 ] Apache Spark commented on SPARK-36713: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-36713) Document new syntax for specifying index type

[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36713: Assignee: Apache Spark > Document new syntax for specifying index type >

[jira] [Assigned] (SPARK-36713) Document new syntax for specifying index type

[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36713: Assignee: (was: Apache Spark) > Document new syntax for specifying index type >

[jira] [Commented] (SPARK-36713) Document new syntax for specifying index type

[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425346#comment-17425346 ] Apache Spark commented on SPARK-36713: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-36304) Refactor fifteenth set of 20 query execution errors to use error classes

[ https://issues.apache.org/jira/browse/SPARK-36304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425342#comment-17425342 ] Apache Spark commented on SPARK-36304: -- User 'dchvn' has created a pull request for this issue:

[jira] [Commented] (SPARK-36304) Refactor fifteenth set of 20 query execution errors to use error classes

[ https://issues.apache.org/jira/browse/SPARK-36304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425341#comment-17425341 ] Apache Spark commented on SPARK-36304: -- User 'dchvn' has created a pull request for this issue:

[jira] [Commented] (SPARK-36711) Support multi-index in new syntax

[ https://issues.apache.org/jira/browse/SPARK-36711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425338#comment-17425338 ] Apache Spark commented on SPARK-36711: -- User 'dchvn' has created a pull request for this issue:

[jira] [Commented] (SPARK-36930) Support ps.MultiIndex.dtypes

[ https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425335#comment-17425335 ] Apache Spark commented on SPARK-36930: -- User 'dchvn' has created a pull request for this issue:

[jira] [Updated] (SPARK-33913) Upgrade Kafka to 2.8.0

[ https://issues.apache.org/jira/browse/SPARK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33913: -- Fix Version/s: 3.2.0 > Upgrade Kafka to 2.8.0 > -- > >

[jira] [Commented] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly

[ https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425331#comment-17425331 ] Apache Spark commented on SPARK-34634: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly

[ https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425332#comment-17425332 ] Apache Spark commented on SPARK-34634: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-36874) Ambiguous Self-Join detected only on right dataframe

[ https://issues.apache.org/jira/browse/SPARK-36874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425330#comment-17425330 ] Apache Spark commented on SPARK-36874: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-36874) Ambiguous Self-Join detected only on right dataframe

[ https://issues.apache.org/jira/browse/SPARK-36874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425329#comment-17425329 ] Apache Spark commented on SPARK-36874: -- User 'sarutak' has created a pull request for this issue:

[jira] [Resolved] (SPARK-36742) Fix ps.to_datetime with plurals of keys like years, months, days

[ https://issues.apache.org/jira/browse/SPARK-36742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36742. -- Fix Version/s: 3.3.0 Assignee: dch nguyen Resolution: Fixed Fixed in

[jira] [Resolved] (SPARK-36918) unionByName shouldn't consider types when comparing structs

[ https://issues.apache.org/jira/browse/SPARK-36918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36918. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34166

[jira] [Resolved] (SPARK-36884) Inline type hints for python/pyspark/sql/session.py

[ https://issues.apache.org/jira/browse/SPARK-36884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36884. -- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in

[jira] [Assigned] (SPARK-36918) unionByName shouldn't consider types when comparing structs

[ https://issues.apache.org/jira/browse/SPARK-36918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36918: --- Assignee: Adam Binford > unionByName shouldn't consider types when comparing structs >

[jira] [Created] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py

dch nguyen created SPARK-36945: -- Summary: Inline type hints for python/pyspark/sql/udf.py Key: SPARK-36945 URL: https://issues.apache.org/jira/browse/SPARK-36945 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py

[ https://issues.apache.org/jira/browse/SPARK-36945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425301#comment-17425301 ] dch nguyen commented on SPARK-36945: working on this > Inline type hints for

[jira] [Commented] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi

[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425299#comment-17425299 ] Apache Spark commented on SPARK-36944: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi

[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36944: Assignee: (was: Apache Spark) > Remove unused python/pyspark/sql/__init__.pyi >

[jira] [Commented] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi

[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425298#comment-17425298 ] Apache Spark commented on SPARK-36944: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi

[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36944: Assignee: Apache Spark > Remove unused python/pyspark/sql/__init__.pyi >

[jira] [Commented] (SPARK-36858) Spark API to apply same function to multiple columns

[ https://issues.apache.org/jira/browse/SPARK-36858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425297#comment-17425297 ] Hyukjin Kwon commented on SPARK-36858: -- you could use var. e.g.) {code} var df = ...

[jira] [Created] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi

dch nguyen created SPARK-36944: -- Summary: Remove unused python/pyspark/sql/__init__.pyi Key: SPARK-36944 URL: https://issues.apache.org/jira/browse/SPARK-36944 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36939: - Fix Version/s: (was: 3.2.0) 3.2.1 > Add orphan migration page into list

[jira] [Assigned] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36939: Assignee: Hyukjin Kwon > Add orphan migration page into list in PySpark documentation >

[jira] [Resolved] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36939. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34195

[jira] [Commented] (SPARK-36943) Improve error message for missing column

[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425290#comment-17425290 ] Apache Spark commented on SPARK-36943: -- User 'karenfeng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36943) Improve error message for missing column

[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36943: Assignee: Apache Spark > Improve error message for missing column >

[jira] [Commented] (SPARK-36943) Improve error message for missing column

[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425289#comment-17425289 ] Apache Spark commented on SPARK-36943: -- User 'karenfeng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36943) Improve error message for missing column

[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36943: Assignee: (was: Apache Spark) > Improve error message for missing column >

[jira] [Created] (SPARK-36943) Improve error message for missing column

Karen Feng created SPARK-36943: -- Summary: Improve error message for missing column Key: SPARK-36943 URL: https://issues.apache.org/jira/browse/SPARK-36943 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-36114) Support subqueries with correlated non-equality predicates

[ https://issues.apache.org/jira/browse/SPARK-36114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425275#comment-17425275 ] Allison Wang commented on SPARK-36114: -- Support non-equality predicates can lift the restrictions

[jira] [Commented] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425213#comment-17425213 ] Xinrong Meng commented on SPARK-36942: -- I'm working on that. > Inline type hints for

[jira] [Created] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

Xinrong Meng created SPARK-36942: Summary: Inline type hints for python/pyspark/sql/readwriter.py Key: SPARK-36942 URL: https://issues.apache.org/jira/browse/SPARK-36942 Project: Spark Issue

[jira] [Commented] (SPARK-35520) Spark-SQL test fails on IBM Z for certain config combinations.

[ https://issues.apache.org/jira/browse/SPARK-35520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425205#comment-17425205 ] Kun Lu commented on SPARK-35520: I've also observed this issue on Spark v3.1.2 on IBM Z. Any comments

[jira] [Assigned] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36941: Assignee: Apache Spark (was: Max Gekk) > Check saving of a dataframe with ANSI

[jira] [Assigned] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36941: Assignee: Max Gekk (was: Apache Spark) > Check saving of a dataframe with ANSI

[jira] [Assigned] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36941: Assignee: Apache Spark (was: Max Gekk) > Check saving of a dataframe with ANSI

[jira] [Commented] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425201#comment-17425201 ] Apache Spark commented on SPARK-36941: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425202#comment-17425202 ] Apache Spark commented on SPARK-36941: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425196#comment-17425196 ] Max Gekk commented on SPARK-36941: -- I am working on this. > Check saving of a dataframe with ANSI

[jira] [Created] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

Max Gekk created SPARK-36941: Summary: Check saving of a dataframe with ANSI intervals to a Hive parquet table Key: SPARK-36941 URL: https://issues.apache.org/jira/browse/SPARK-36941 Project: Spark

[jira] [Commented] (SPARK-32952) Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1

[ https://issues.apache.org/jira/browse/SPARK-32952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425187#comment-17425187 ] Kun Lu commented on SPARK-32952: This issue still exists on Spark v3.1.2 on IBM Z. Any updates from the

[jira] [Commented] (SPARK-32929) StreamSuite failure on IBM Z: - SPARK-20432: union one stream with itself

[ https://issues.apache.org/jira/browse/SPARK-32929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425179#comment-17425179 ] Kun Lu commented on SPARK-32929: This issue has been addressed in commit 

[jira] [Commented] (SPARK-36936) spark-hadoop-cloud broken on release and only published via 3rd party repositories

[ https://issues.apache.org/jira/browse/SPARK-36936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425162#comment-17425162 ] Chao Sun commented on SPARK-36936: -- [~colin.williams] which version of {{spark-hadoop-cloud}} you were

[jira] [Resolved] (SPARK-36937) Change OrcSourceSuite to test both V1 and V2 sources.

[ https://issues.apache.org/jira/browse/SPARK-36937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36937. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34194

[jira] [Assigned] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36940: Assignee: (was: Apache Spark) > Inline type hints for

[jira] [Commented] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425143#comment-17425143 ] Apache Spark commented on SPARK-36940: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36940: Assignee: Apache Spark > Inline type hints for python/pyspark/sql/avro/functions.py >

[jira] [Commented] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level

[ https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425141#comment-17425141 ] Apache Spark commented on SPARK-36935: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level

[ https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36935: Assignee: (was: Apache Spark) > Enhance ParquetSchemaConverter to capture Parquet

[jira] [Commented] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level

[ https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425140#comment-17425140 ] Apache Spark commented on SPARK-36935: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level

[ https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36935: Assignee: Apache Spark > Enhance ParquetSchemaConverter to capture Parquet repetition &

[jira] [Updated] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36940: - Description: Inline type hints for python/pyspark/sql/avro/functions.py.   Currently, we use

[jira] [Created] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

Xinrong Meng created SPARK-36940: Summary: Inline type hints for python/pyspark/sql/avro/functions.py Key: SPARK-36940 URL: https://issues.apache.org/jira/browse/SPARK-36940 Project: Spark

[jira] [Commented] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425132#comment-17425132 ] Xinrong Meng commented on SPARK-36940: -- I'm working on this. > Inline type hints for

[jira] [Assigned] (SPARK-36300) Refactor eleventh set of 20 query execution errors to use error classes

[ https://issues.apache.org/jira/browse/SPARK-36300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36300: Assignee: (was: Apache Spark) > Refactor eleventh set of 20 query execution errors

[jira] [Assigned] (SPARK-36300) Refactor eleventh set of 20 query execution errors to use error classes

[ https://issues.apache.org/jira/browse/SPARK-36300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36300: Assignee: Apache Spark > Refactor eleventh set of 20 query execution errors to use error

[jira] [Commented] (SPARK-36300) Refactor eleventh set of 20 query execution errors to use error classes

[ https://issues.apache.org/jira/browse/SPARK-36300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425129#comment-17425129 ] Apache Spark commented on SPARK-36300: -- User 'changvvb' has created a pull request for this issue:

[jira] [Commented] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425072#comment-17425072 ] Gengliang Wang commented on SPARK-36905: [~shardulm] Thanks for reporting the issue. I don't

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425016#comment-17425016 ] Mridul Muralidharan commented on SPARK-36892: - Sounds good [~Gengliang.Wang], I am not aware

[jira] [Commented] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424990#comment-17424990 ] Apache Spark commented on SPARK-36938: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36938: Assignee: (was: Apache Spark) > Inline type hints for group.py in python/pyspark/sql

[jira] [Commented] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424989#comment-17424989 ] Apache Spark commented on SPARK-36938: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36938: Assignee: Apache Spark > Inline type hints for group.py in python/pyspark/sql >

[jira] [Commented] (SPARK-36858) Spark API to apply same function to multiple columns

[ https://issues.apache.org/jira/browse/SPARK-36858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424988#comment-17424988 ] Armand BERGES commented on SPARK-36858: --- [~hyukjin.kwon] How would you do this ? >From my point,

[jira] [Commented] (SPARK-36751) octet_length/bit_length API is not implemented on Scala/Python/R

[ https://issues.apache.org/jira/browse/SPARK-36751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424939#comment-17424939 ] Apache Spark commented on SPARK-36751: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Resolved] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36934. -- Resolution: Not A Problem > Timestamp are written as array bytes. >

[jira] [Assigned] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36939: Assignee: Apache Spark > Add orphan migration page into list in PySpark documentation >

[jira] [Commented] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424935#comment-17424935 ] Apache Spark commented on SPARK-36939: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36939: Assignee: (was: Apache Spark) > Add orphan migration page into list in PySpark

[jira] [Updated] (SPARK-36939) Add orphan migration page into list in PySpark documentation

[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36939: - Issue Type: Documentation (was: Test) > Add orphan migration page into list in PySpark

[jira] [Created] (SPARK-36939) Add orphan migration page into list in PySpark documentation

Hyukjin Kwon created SPARK-36939: Summary: Add orphan migration page into list in PySpark documentation Key: SPARK-36939 URL: https://issues.apache.org/jira/browse/SPARK-36939 Project: Spark

[jira] [Updated] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-36938: --- Summary: Inline type hints for group.py in python/pyspark/sql (was: nline type hints for

[jira] [Created] (SPARK-36938) nline type hints for group.py in python/pyspark/sql

dch nguyen created SPARK-36938: -- Summary: nline type hints for group.py in python/pyspark/sql Key: SPARK-36938 URL: https://issues.apache.org/jira/browse/SPARK-36938 Project: Spark Issue

[jira] [Commented] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424929#comment-17424929 ] dch nguyen commented on SPARK-36938: i am working on this > Inline type hints for group.py in

[jira] [Resolved] (SPARK-36887) Inline type hints for python/pyspark/sql/conf.py

[ https://issues.apache.org/jira/browse/SPARK-36887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen resolved SPARK-36887. Resolution: Resolved This issue is resolved by https://issues.apache.org/jira/browse/SPARK-36906

[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424917#comment-17424917 ] Hyukjin Kwon commented on SPARK-36934: -- Looks like Apache Drill only implements TIMESTAMP_MILLIS in

[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424901#comment-17424901 ] Bjørn Jørgensen commented on SPARK-36934: - With .config("spark.sql.parquet.outputTimestampType",

[jira] [Assigned] (SPARK-36919) Make BadRecordException serializable

[ https://issues.apache.org/jira/browse/SPARK-36919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36919: Assignee: Tianhan Hu > Make BadRecordException serializable >

[jira] [Resolved] (SPARK-36919) Make BadRecordException serializable

[ https://issues.apache.org/jira/browse/SPARK-36919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36919. -- Fix Version/s: 3.1.3 3.2.0 3.0.4 Resolution:

[jira] [Resolved] (SPARK-36927) Inline type hints for python/pyspark/sql/window.py

[ https://issues.apache.org/jira/browse/SPARK-36927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36927. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34173

[jira] [Assigned] (SPARK-36927) Inline type hints for python/pyspark/sql/window.py

[ https://issues.apache.org/jira/browse/SPARK-36927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36927: Assignee: Xinrong Meng > Inline type hints for python/pyspark/sql/window.py >

[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424885#comment-17424885 ] Hyukjin Kwon commented on SPARK-36934: -- what about TIMESTAMP_MILLIS? > Timestamp are written as

[jira] [Updated] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-36934: Component/s: Spark Core > Timestamp are written as array bytes. >

[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424860#comment-17424860 ] Bjørn Jørgensen commented on SPARK-36934: - .config("spark.sql.parquet.outputTimestampType",

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424825#comment-17424825 ] Gengliang Wang commented on SPARK-36892: [~mridulm80] [~mshen] [~zhouyejoe] [~apatnam] Again,

[jira] [Resolved] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36892. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34156

[jira] [Assigned] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-36892: -- Assignee: Ye Zhou > Disable batch fetch for a shuffle when push based shuffle is

[jira] [Commented] (SPARK-36917) java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in i

[ https://issues.apache.org/jira/browse/SPARK-36917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424811#comment-17424811 ] Hyukjin Kwon commented on SPARK-36917: -- i am not actually aware of an issue related to this either.

[jira] [Updated] (SPARK-36917) java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in ins

[ https://issues.apache.org/jira/browse/SPARK-36917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36917: - Target Version/s: (was: 3.1.2) > java.lang.ClassCastException: cannot assign instance of >

[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424806#comment-17424806 ] Hyukjin Kwon commented on SPARK-36934: -- I think this is an issue in Spark. Can you try with

[jira] [Updated] (SPARK-36934) Timestamp are written as array bytes.

[ https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36934: - Description: This is tested with master build 04.10.21 {code} df = ps.DataFrame({'year':

[jira] [Commented] (SPARK-36936) spark-hadoop-cloud broken on release and only published via 3rd party repositories

[ https://issues.apache.org/jira/browse/SPARK-36936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424803#comment-17424803 ] Hyukjin Kwon commented on SPARK-36936: -- cc [~sunchao] FYI > spark-hadoop-cloud broken on release

[jira] [Resolved] (SPARK-36930) Support ps.MultiIndex.dtypes

[ https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36930. -- Fix Version/s: 3.3.0 Assignee: dch nguyen Resolution: Fixed Fixed in

[jira] [Assigned] (SPARK-36937) Change OrcSourceSuite to test both V1 and V2 sources.

[ https://issues.apache.org/jira/browse/SPARK-36937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36937: Assignee: Apache Spark (was: Kousuke Saruta) > Change OrcSourceSuite to test both V1

[jira] [Assigned] (SPARK-36937) Change OrcSourceSuite to test both V1 and V2 sources.

[ https://issues.apache.org/jira/browse/SPARK-36937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36937: Assignee: Kousuke Saruta (was: Apache Spark) > Change OrcSourceSuite to test both V1

[jira] [Commented] (SPARK-36937) Change OrcSourceSuite to test both V1 and V2 sources.

[ https://issues.apache.org/jira/browse/SPARK-36937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424799#comment-17424799 ] Apache Spark commented on SPARK-36937: -- User 'sarutak' has created a pull request for this issue:

  1   2   >