[jira] [Resolved] (SPARK-48705) Explicitly use worker_main when it starts with pyspark

[ https://issues.apache.org/jira/browse/SPARK-48705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48705. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47077 [https://gi

[jira] [Assigned] (SPARK-48705) Explicitly use worker_main when it starts with pyspark

[ https://issues.apache.org/jira/browse/SPARK-48705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48705: Assignee: Hyukjin Kwon > Explicitly use worker_main when it starts with pyspark > ---

[jira] [Assigned] (SPARK-48706) Python UDF in higher order functions should not throw internal error

[ https://issues.apache.org/jira/browse/SPARK-48706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48706: Assignee: Hyukjin Kwon > Python UDF in higher order functions should not throw internal error > -

[jira] [Resolved] (SPARK-48706) Python UDF in higher order functions should not throw internal error

[ https://issues.apache.org/jira/browse/SPARK-48706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48706. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47079 [https://github.com

[jira] [Updated] (SPARK-48719) Wrong Result in regr_slope®r_intercept Aggregate with Tuples has NULL

[ https://issues.apache.org/jira/browse/SPARK-48719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathon Lee updated SPARK-48719: - Description: When calculate slope and intercept using regr_slope & regr_intercept aggregate: (u

[jira] [Created] (SPARK-48719) Wrong Result in regr_slope®r_intercept Aggregate with Tuples has NULL

Jonathon Lee created SPARK-48719: Summary: Wrong Result in regr_slope®r_intercept Aggregate with Tuples has NULL Key: SPARK-48719 URL: https://issues.apache.org/jira/browse/SPARK-48719 Project: Spark

[jira] [Resolved] (SPARK-48573) Upgrade ICU version

[ https://issues.apache.org/jira/browse/SPARK-48573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48573. -- Target Version/s: 4.0.0 Assignee: Mihailo Milosevic Resolution: Fixed > Upgrade

[jira] [Updated] (SPARK-48573) Upgrade ICU version

[ https://issues.apache.org/jira/browse/SPARK-48573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-48573: - Fix Version/s: 4.0.0 > Upgrade ICU version > --- > > Key: SPARK-48573 >

[jira] [Resolved] (SPARK-48718) Got incastable error when deserializer in cogroup is resolved during application of DeduplicateRelation rule

[ https://issues.apache.org/jira/browse/SPARK-48718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48718. - Fix Version/s: 4.0.0 Assignee: Xinyi Yu Resolution: Fixed > Got incastable error

[jira] [Updated] (SPARK-43781) IllegalStateException when cogrouping two datasets derived from the same source

[ https://issues.apache.org/jira/browse/SPARK-43781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-43781: Fix Version/s: 4.0.0 (was: 3.5.0) > IllegalStateException when cogrouping t

[jira] [Created] (SPARK-48718) Got incastable error when deserializer in cogroup is resolved during application of DeduplicateRelation rule

Xinyi Yu created SPARK-48718: Summary: Got incastable error when deserializer in cogroup is resolved during application of DeduplicateRelation rule Key: SPARK-48718 URL: https://issues.apache.org/jira/browse/SPARK-487

[jira] [Created] (SPARK-48717) Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries

Wei Liu created SPARK-48717: --- Summary: Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries Key: SPARK-48717 URL: https://issues.apache.org/jira

[jira] [Resolved] (SPARK-48638) Native QueryExecution information for the dataframe

[ https://issues.apache.org/jira/browse/SPARK-48638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48638. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46996 [https://gi

[jira] [Assigned] (SPARK-48638) Native QueryExecution information for the dataframe

[ https://issues.apache.org/jira/browse/SPARK-48638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48638: Assignee: Martin Grund > Native QueryExecution information for the dataframe > --

[jira] [Created] (SPARK-48716) Add JobGroupId to onSqlStart

Lingkai Kong created SPARK-48716: Summary: Add JobGroupId to onSqlStart Key: SPARK-48716 URL: https://issues.apache.org/jira/browse/SPARK-48716 Project: Spark Issue Type: Task Compo

[jira] (SPARK-39901) Reconsider design of ignoreCorruptFiles feature

[ https://issues.apache.org/jira/browse/SPARK-39901 ] Wei Guo deleted comment on SPARK-39901: - was (Author: wayne guo): The `ignoreCorruptFiles` features in SQL(spark.sql.files.ignoreCorruptFiles) and RDD(spark.files.ignoreCorruptFiles) scenarios n

[jira] [Created] (SPARK-48715) UTF8String - Java String conversions should use Unicode replacement logic

Uroš Bojanić created SPARK-48715: Summary: UTF8String - Java String conversions should use Unicode replacement logic Key: SPARK-48715 URL: https://issues.apache.org/jira/browse/SPARK-48715 Project: Sp

[jira] [Resolved] (SPARK-48578) Add new expressions for UTF8 string validation

[ https://issues.apache.org/jira/browse/SPARK-48578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48578. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46845 [https://gith

[jira] [Assigned] (SPARK-48578) Add new expressions for UTF8 string validation

[ https://issues.apache.org/jira/browse/SPARK-48578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48578: --- Assignee: Uroš Bojanić > Add new expressions for UTF8 string validation > -

[jira] [Commented] (SPARK-47927) Nullability after join not respected in UDF

[ https://issues.apache.org/jira/browse/SPARK-47927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17859940#comment-17859940 ] GridGain Integration commented on SPARK-47927: -- User 'cloud-fan' has create

[jira] [Created] (SPARK-48714) Implement df.mergeInto in PySpark

Pengfei Xu created SPARK-48714: -- Summary: Implement df.mergeInto in PySpark Key: SPARK-48714 URL: https://issues.apache.org/jira/browse/SPARK-48714 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-48713) Add index range check for UnsafeRow.pointTo when baseObject is byte array

wuyi created SPARK-48713: Summary: Add index range check for UnsafeRow.pointTo when baseObject is byte array Key: SPARK-48713 URL: https://issues.apache.org/jira/browse/SPARK-48713 Project: Spark Is

[jira] [Updated] (SPARK-48712) Perf Improvement for Encode with empty string and UTF-8 charset

[ https://issues.apache.org/jira/browse/SPARK-48712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-48712: - Description:  Apple M2 Max  encode:                                   Best Time(ms)   Avg Time(ms)   St

[jira] [Created] (SPARK-48712) Perf Improvement for Encode with empty string and UTF-8 charset

Kent Yao created SPARK-48712: Summary: Perf Improvement for Encode with empty string and UTF-8 charset Key: SPARK-48712 URL: https://issues.apache.org/jira/browse/SPARK-48712 Project: Spark Issu

[jira] [Resolved] (SPARK-48693) simplify and unify toString of Invoke and StaticInvoke

[ https://issues.apache.org/jira/browse/SPARK-48693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48693. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47066 [https://github.com

[jira] [Created] (SPARK-48711) OOM killer may leave SparkContext in broken state causing ConnectionRefusedError

Rafal Wojdyla created SPARK-48711: - Summary: OOM killer may leave SparkContext in broken state causing ConnectionRefusedError Key: SPARK-48711 URL: https://issues.apache.org/jira/browse/SPARK-48711 Pr

[jira] [Created] (SPARK-48710) Incompatibilities with NumPy 2.0

Patrick Marx created SPARK-48710: Summary: Incompatibilities with NumPy 2.0 Key: SPARK-48710 URL: https://issues.apache.org/jira/browse/SPARK-48710 Project: Spark Issue Type: Bug Co

[jira] [Assigned] (SPARK-48698) Support analyze column stats for tables with collated columns

[ https://issues.apache.org/jira/browse/SPARK-48698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48698: -- Assignee: (was: Apache Spark) > Support analyze column stats for tables with coll

[jira] [Assigned] (SPARK-48698) Support analyze column stats for tables with collated columns

[ https://issues.apache.org/jira/browse/SPARK-48698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48698: -- Assignee: (was: Apache Spark) > Support analyze column stats for tables with coll

[jira] [Assigned] (SPARK-48177) Upgrade `Parquet` to 1.14.1

[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48177: -- Assignee: Apache Spark (was: Fokko Driesprong) > Upgrade `Parquet` to 1.14.1 > -

[jira] [Created] (SPARK-48709) varchar resolution mismatch for DataSourceV2 CTAS

Yuming Wang created SPARK-48709: --- Summary: varchar resolution mismatch for DataSourceV2 CTAS Key: SPARK-48709 URL: https://issues.apache.org/jira/browse/SPARK-48709 Project: Spark Issue Type: B