[jira] [Commented] (SPARK-34276) Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12

[ https://issues.apache.org/jira/browse/SPARK-34276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425959#comment-17425959 ] Micah Kornfield commented on SPARK-34276: - Sorry for the late reply.  PARQUET-2089 has been a

[jira] [Updated] (SPARK-35531) Can not insert into hive bucket table if create table with upper case schema

[ https://issues.apache.org/jira/browse/SPARK-35531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-35531: --- Affects Version/s: 3.0.0 3.1.1 > Can not insert into hive bucket

[jira] [Commented] (SPARK-35531) Can not insert into hive bucket table if create table with upper case schema

[ https://issues.apache.org/jira/browse/SPARK-35531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425954#comment-17425954 ] Gengliang Wang commented on SPARK-35531: I can reproduce the issue on 3.0.0 and 3.1.1. It's a

[jira] [Commented] (SPARK-36952) Inline type hints for python/pyspark/resource/information.py and python/pyspark/resource/profile.py

[ https://issues.apache.org/jira/browse/SPARK-36952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425949#comment-17425949 ] dch nguyen commented on SPARK-36952: working on this > Inline type hints for

[jira] [Commented] (SPARK-36952) Inline type hints for python/pyspark/resource/information.py and python/pyspark/resource/profile.py

[ https://issues.apache.org/jira/browse/SPARK-36952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425947#comment-17425947 ] dgd_contributor commented on SPARK-36952: - working on this > Inline type hints for

[jira] [Issue Comment Deleted] (SPARK-36952) Inline type hints for python/pyspark/resource/information.py and python/pyspark/resource/profile.py

[ https://issues.apache.org/jira/browse/SPARK-36952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dgd_contributor updated SPARK-36952: Comment: was deleted (was: working on this) > Inline type hints for

[jira] [Commented] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425945#comment-17425945 ] Apache Spark commented on SPARK-36953: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36953: Assignee: (was: Apache Spark) > Expose SQL state and error class in PySpark

[jira] [Assigned] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36953: Assignee: Apache Spark > Expose SQL state and error class in PySpark exceptions >

[jira] [Commented] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425944#comment-17425944 ] Apache Spark commented on SPARK-36953: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

Hyukjin Kwon created SPARK-36953: Summary: Expose SQL state and error class in PySpark exceptions Key: SPARK-36953 URL: https://issues.apache.org/jira/browse/SPARK-36953 Project: Spark Issue

[jira] [Commented] (SPARK-35531) Can not insert into hive bucket table if create table with upper case schema

[ https://issues.apache.org/jira/browse/SPARK-35531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425932#comment-17425932 ] Apache Spark commented on SPARK-35531: -- User 'AngersZh' has created a pull request for this

[jira] [Created] (SPARK-36952) Inline type hints for python/pyspark/resource/information.py and python/pyspark/resource/profile.py

dgd_contributor created SPARK-36952: --- Summary: Inline type hints for python/pyspark/resource/information.py and python/pyspark/resource/profile.py Key: SPARK-36952 URL:

[jira] [Updated] (SPARK-36903) oom exception occurred during code generation due to a large number of case when branches

[ https://issues.apache.org/jira/browse/SPARK-36903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-36903: --- Description: I have a spark task that contains many case when branches. When I run it, the driver

[jira] [Updated] (SPARK-36903) oom exception occurred during code generation due to a large number of case when branches

[ https://issues.apache.org/jira/browse/SPARK-36903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-36903: --- Description: I have a spark task that contains many case when branches. When I run it, the driver

[jira] [Commented] (SPARK-36839) Add daily build with Hadoop 2 profile in GitHub Actions build

[ https://issues.apache.org/jira/browse/SPARK-36839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425879#comment-17425879 ] Apache Spark commented on SPARK-36839: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-36839) Add daily build with Hadoop 2 profile in GitHub Actions build

[ https://issues.apache.org/jira/browse/SPARK-36839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425878#comment-17425878 ] Apache Spark commented on SPARK-36839: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-36950) Normalize semi-structured data into a flat table.

[ https://issues.apache.org/jira/browse/SPARK-36950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425873#comment-17425873 ] Hyukjin Kwon commented on SPARK-36950: -- Thanks [~bjornjorgensen] > Normalize semi-structured data

[jira] [Updated] (SPARK-36947) Exception when trying to access Row field using getAs method

[ https://issues.apache.org/jira/browse/SPARK-36947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36947: - Priority: Major (was: Blocker) > Exception when trying to access Row field using getAs method

[jira] [Updated] (SPARK-36950) Normalize semi-structured data into a flat table.

[ https://issues.apache.org/jira/browse/SPARK-36950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36950: - Issue Type: Improvement (was: Wish) > Normalize semi-structured data into a flat table. >

[jira] [Resolved] (SPARK-29871) Flaky test: ImageFileFormatTest.test_read_images

[ https://issues.apache.org/jira/browse/SPARK-29871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29871. -- Fix Version/s: 3.3.0 Assignee: Hyukjin Kwon Resolution: Fixed > Flaky test:

[jira] [Commented] (SPARK-29871) Flaky test: ImageFileFormatTest.test_read_images

[ https://issues.apache.org/jira/browse/SPARK-29871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425869#comment-17425869 ] Hyukjin Kwon commented on SPARK-29871: -- Fixed in https://github.com/apache/spark/pull/34187 >

[jira] [Created] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

Xinrong Meng created SPARK-36951: Summary: Inline type hints for python/pyspark/sql/column.py Key: SPARK-36951 URL: https://issues.apache.org/jira/browse/SPARK-36951 Project: Spark Issue

[jira] [Commented] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

[ https://issues.apache.org/jira/browse/SPARK-36951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425794#comment-17425794 ] Xinrong Meng commented on SPARK-36951: -- I am working on this. > Inline type hints for

[jira] [Created] (SPARK-36950) Normalize semi-structured data into a flat table.

Bjørn Jørgensen created SPARK-36950: --- Summary: Normalize semi-structured data into a flat table. Key: SPARK-36950 URL: https://issues.apache.org/jira/browse/SPARK-36950 Project: Spark

[jira] [Commented] (SPARK-36936) spark-hadoop-cloud broken on release and only published via 3rd party repositories

[ https://issues.apache.org/jira/browse/SPARK-36936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425764#comment-17425764 ] Colin Williams commented on SPARK-36936: [~csun] when I see SPARK-35844 I see 3.2.0 version for

[jira] [Created] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

Max Gekk created SPARK-36949: Summary: Fix CREATE TABLE AS SELECT of ANSI intervals Key: SPARK-36949 URL: https://issues.apache.org/jira/browse/SPARK-36949 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36940. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Commented] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425733#comment-17425733 ] Apache Spark commented on SPARK-36942: -- User 'xinrong-databricks' has created a pull request for

[jira] [Commented] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425732#comment-17425732 ] Apache Spark commented on SPARK-36942: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36942: Assignee: Apache Spark > Inline type hints for python/pyspark/sql/readwriter.py >

[jira] [Assigned] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36942: Assignee: (was: Apache Spark) > Inline type hints for

[jira] [Assigned] (SPARK-36948) Check CREATE TABLE with ANSI intervals using Hive external catalog and Parquet

[ https://issues.apache.org/jira/browse/SPARK-36948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36948: Assignee: Max Gekk (was: Apache Spark) > Check CREATE TABLE with ANSI intervals using

[jira] [Commented] (SPARK-36948) Check CREATE TABLE with ANSI intervals using Hive external catalog and Parquet

[ https://issues.apache.org/jira/browse/SPARK-36948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425727#comment-17425727 ] Apache Spark commented on SPARK-36948: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36948) Check CREATE TABLE with ANSI intervals using Hive external catalog and Parquet

[ https://issues.apache.org/jira/browse/SPARK-36948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36948: Assignee: Apache Spark (was: Max Gekk) > Check CREATE TABLE with ANSI intervals using

[jira] [Commented] (SPARK-36948) Check CREATE TABLE with ANSI intervals using Hive external catalog and Parquet

[ https://issues.apache.org/jira/browse/SPARK-36948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425726#comment-17425726 ] Apache Spark commented on SPARK-36948: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Created] (SPARK-36948) Check CREATE TABLE with ANSI intervals using Hive external catalog and Parquet

Max Gekk created SPARK-36948: Summary: Check CREATE TABLE with ANSI intervals using Hive external catalog and Parquet Key: SPARK-36948 URL: https://issues.apache.org/jira/browse/SPARK-36948 Project:

[jira] [Updated] (SPARK-36947) Exception when trying to access Row field using getAs method

[ https://issues.apache.org/jira/browse/SPARK-36947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandros Mavrommatis updated SPARK-36947: --- Description: I have an input dataframe *df* with the following schema:

[jira] [Created] (SPARK-36947) Exception when trying to access Row field using getAs method

Alexandros Mavrommatis created SPARK-36947: -- Summary: Exception when trying to access Row field using getAs method Key: SPARK-36947 URL: https://issues.apache.org/jira/browse/SPARK-36947

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425570#comment-17425570 ] Apache Spark commented on SPARK-36900: -- User 'srowen' has created a pull request for this issue:

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425568#comment-17425568 ] Apache Spark commented on SPARK-36900: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: Apache Spark > "SPARK-36464: size returns correct positive number even with

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: (was: Apache Spark) > "SPARK-36464: size returns correct positive number

[jira] [Updated] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-36900: - Priority: Minor (was: Major) > "SPARK-36464: size returns correct positive number even with

[jira] [Assigned] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36798: --- Assignee: Harsh Panchal > When SparkContext is stopped, metrics system

[jira] [Resolved] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36798. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Commented] (SPARK-36396) Implement DataFrame.cov

[ https://issues.apache.org/jira/browse/SPARK-36396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425510#comment-17425510 ] Apache Spark commented on SPARK-36396: -- User 'dchvn' has created a pull request for this issue:

[jira] [Commented] (SPARK-36402) Implement Series.combine

[ https://issues.apache.org/jira/browse/SPARK-36402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425493#comment-17425493 ] Apache Spark commented on SPARK-36402: -- User 'dchvn' has created a pull request for this issue:

[jira] [Commented] (SPARK-36946) Support time for ps.to_datetime

[ https://issues.apache.org/jira/browse/SPARK-36946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425473#comment-17425473 ] Apache Spark commented on SPARK-36946: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36946) Support time for ps.to_datetime

[ https://issues.apache.org/jira/browse/SPARK-36946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36946: Assignee: Apache Spark > Support time for ps.to_datetime >

[jira] [Assigned] (SPARK-36946) Support time for ps.to_datetime

[ https://issues.apache.org/jira/browse/SPARK-36946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36946: Assignee: (was: Apache Spark) > Support time for ps.to_datetime >

[jira] [Created] (SPARK-36946) Support time for ps.to_datetime

dgd_contributor created SPARK-36946: --- Summary: Support time for ps.to_datetime Key: SPARK-36946 URL: https://issues.apache.org/jira/browse/SPARK-36946 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-36707) Support to specify index type and name in pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-36707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36707. -- Fix Version/s: 3.3.0 Assignee: Hyukjin Kwon Resolution: Done > Support to

[jira] [Resolved] (SPARK-36713) Document new syntax for specifying index type

[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36713. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34210

[jira] [Assigned] (SPARK-36713) Document new syntax for specifying index type

[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36713: Assignee: Hyukjin Kwon > Document new syntax for specifying index type >

[jira] [Updated] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly

[ https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34634: - Fix Version/s: 3.1.3 > Self-join with script transformation failed to resolve attribute

[jira] [Updated] (SPARK-36874) Ambiguous Self-Join detected only on right dataframe

[ https://issues.apache.org/jira/browse/SPARK-36874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36874: - Fix Version/s: 3.1.3 > Ambiguous Self-Join detected only on right dataframe >

[jira] [Resolved] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table

[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36941. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34201