[jira] [Resolved] (SPARK-48585) Make `JdbcDialect.classifyException` throw out the original exception

2024-06-17 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-48585. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46937 [https://github.com

[jira] [Assigned] (SPARK-48585) Make `JdbcDialect.classifyException` throw out the original exception

2024-06-17 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-48585: Assignee: BingKun Pan > Make `JdbcDialect.classifyException` throw out the original exception > -

[jira] [Resolved] (SPARK-48647) Refine the error message for YearMonthIntervalType in df.collect

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48647. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47004 [https://gi

[jira] [Updated] (SPARK-48649) Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48649: --- Labels: pull-request-available (was: ) > Add "ignoreInvalidPartitionPaths" and > "spark.sq

[jira] [Resolved] (SPARK-48644) Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex

2024-06-17 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48644. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47001 [https://github.com

[jira] [Assigned] (SPARK-48648) Make tags properly threadlocal

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48648: Assignee: Hyukjin Kwon > Make tags properly threadlocal > --

[jira] [Resolved] (SPARK-48648) Make tags properly threadlocal

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48648. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47005 [https://gi

[jira] [Resolved] (SPARK-47910) Memory leak when interrupting shuffle write using zstd compression

2024-06-17 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-47910. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue res

[jira] [Assigned] (SPARK-47910) Memory leak when interrupting shuffle write using zstd compression

2024-06-17 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-47910: --- Assignee: JacobZheng > Memory leak when interrupting shuffle write using zs

[jira] [Updated] (SPARK-48649) Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths

2024-06-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-48649: - Description: When having a table directory with invalid partitions such as: {code:java} table/

[jira] [Created] (SPARK-48649) Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths

2024-06-17 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-48649: Summary: Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths Key: SPARK-48649 URL: https://issue

[jira] [Updated] (SPARK-48648) Make tags properly threadlocal

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48648: --- Labels: pull-request-available (was: ) > Make tags properly threadlocal > -

[jira] [Created] (SPARK-48648) Make tags properly threadlocal

2024-06-17 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-48648: Summary: Make tags properly threadlocal Key: SPARK-48648 URL: https://issues.apache.org/jira/browse/SPARK-48648 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-48647) Refine the error message for YearMonthIntervalType in df.collect

2024-06-17 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-48647: - Summary: Refine the error message for YearMonthIntervalType in df.collect Key: SPARK-48647 URL: https://issues.apache.org/jira/browse/SPARK-48647 Project: Spark

[jira] [Resolved] (SPARK-48643) Narrow down default regex of sql options redaction.

2024-06-17 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Wang resolved SPARK-48643. --- Resolution: Won't Fix > Narrow down default regex of sql options redaction. > --

[jira] [Assigned] (SPARK-48497) Add user guide for batch data source write API

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48497: Assignee: Allison Wang > Add user guide for batch data source write API > ---

[jira] [Resolved] (SPARK-48497) Add user guide for batch data source write API

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48497. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46833 [https://gi

[jira] [Assigned] (SPARK-48567) Pyspark StreamingQuery lastProgress and friend should return actual StreamingQueryProgress

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48567: Assignee: Wei Liu > Pyspark StreamingQuery lastProgress and friend should return actual

[jira] [Resolved] (SPARK-48567) Pyspark StreamingQuery lastProgress and friend should return actual StreamingQueryProgress

2024-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48567. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46921 [https://gi

[jira] [Commented] (SPARK-48472) Enable reflect expressions with collated strings

2024-06-17 Thread Mihailo Aleksic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855742#comment-17855742 ] Mihailo Aleksic commented on SPARK-48472: - Working on this > Enable reflect exp

[jira] [Commented] (SPARK-48463) MLLib function unable to handle nested data

2024-06-17 Thread Chhavi Bansal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855714#comment-17855714 ] Chhavi Bansal commented on SPARK-48463: --- Thank you [~weichenxu123] for reopening t

[jira] [Resolved] (SPARK-48642) False reported SparkOutOfMemoryError caused by killing task on spilling

2024-06-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48642. --- Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution: Fix

[jira] [Assigned] (SPARK-48642) False reported SparkOutOfMemoryError caused by killing task on spilling

2024-06-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48642: - Assignee: Cheng Pan > False reported SparkOutOfMemoryError caused by killing task on sp

[jira] [Assigned] (SPARK-48566) [Bug] Partition indices are incorrect when UDTF analyze() uses both select and partitionColumns

2024-06-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-48566: - Assignee: Daniel > [Bug] Partition indices are incorrect when UDTF analyze() uses both

[jira] [Resolved] (SPARK-48566) [Bug] Partition indices are incorrect when UDTF analyze() uses both select and partitionColumns

2024-06-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-48566. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46918 [https://

[jira] [Resolved] (SPARK-47579) Spark core: Migrate logInfo with variables to structured logging framework

2024-06-17 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47579. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46951 [https:

[jira] [Updated] (SPARK-48177) Upgrade `Parquet` to 1.14.1

2024-06-17 Thread Fokko Driesprong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated SPARK-48177: - Summary: Upgrade `Parquet` to 1.14.1 (was: Upgrade `Parquet` to 1.14.0) > Upgrade `Parq

[jira] [Resolved] (SPARK-48603) Update *ParquetReadSchemaSuite to cover type widen capability

2024-06-17 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48603. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46959 [https://github.com

[jira] [Assigned] (SPARK-48603) Update *ParquetReadSchemaSuite to cover type widen capability

2024-06-17 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48603: Assignee: Cheng Pan > Update *ParquetReadSchemaSuite to cover type widen capability > ---

[jira] [Resolved] (SPARK-48641) Upgrade curator to 5.7.0

2024-06-17 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48641. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46998 [https://github.com

[jira] [Updated] (SPARK-48644) Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48644: --- Labels: pull-request-available (was: ) > Do a length check and throw COLLECTION_SIZE_LIMIT_

[jira] [Assigned] (SPARK-48620) Fix internal raw data leak in YearMonthIntervalType

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48620: -- Assignee: Apache Spark > Fix internal raw data leak in YearMonthIntervalType > --

[jira] [Assigned] (SPARK-48620) Fix internal raw data leak in YearMonthIntervalType

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48620: -- Assignee: (was: Apache Spark) > Fix internal raw data leak in YearMonthIntervalTy

[jira] [Resolved] (SPARK-48572) Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions

2024-06-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48572. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46943 [https://gith

[jira] [Assigned] (SPARK-48572) Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions

2024-06-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48572: --- Assignee: Mihailo Milosevic > Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindo

[jira] [Updated] (SPARK-48642) False reported SparkOutOfMemoryError caused by killing task on spilling

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48642: --- Labels: pull-request-available (was: ) > False reported SparkOutOfMemoryError caused by kil

[jira] [Updated] (SPARK-48643) Narrow down default regex of sql options redaction.

2024-06-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48643: --- Labels: pull-request-available (was: ) > Narrow down default regex of sql options redaction

[jira] [Resolved] (SPARK-48600) Fix FrameLessOffsetWindowFunction expressions implicit casting

2024-06-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48600. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46987 [https://gith

[jira] [Assigned] (SPARK-48600) Fix FrameLessOffsetWindowFunction expressions implicit casting

2024-06-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48600: --- Assignee: Mihailo Milosevic > Fix FrameLessOffsetWindowFunction expressions implicit castin