[jira] [Assigned] (SPARK-48620) Fix internal raw data leak in YearMonthIntervalType

2024-06-19 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-48620: - Assignee: Ruifeng Zheng > Fix internal raw data leak in YearMonthIntervalType > ---

[jira] [Resolved] (SPARK-48620) Fix internal raw data leak in YearMonthIntervalType

2024-06-19 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48620. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46975 [https://

[jira] [Resolved] (SPARK-48591) Simplify the if-else branches with `F.lit`

2024-06-19 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48591. --- Resolution: Fixed Issue resolved by pull request 47023 [https://github.com/apache/spark/pull

[jira] [Resolved] (SPARK-48664) Add official image Dockerfile for Apache Spark 4.0.0-preview1

2024-06-19 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48664. -- Target Version/s: 4.0.0 Assignee: Wenchen Fan Resolution: Fixed > Add official i

[jira] [Updated] (SPARK-48667) Arrow python UDFS didn't support UDT as output type

2024-06-19 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-48667: -- Description: {code:java} df.select(udf(lambda x: x, returnType=ExamplePointUDT(), useArrow=useArrow)(

[jira] [Updated] (SPARK-48667) Arrow python UDFS didn't support UDT as output type

2024-06-19 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-48667: -- Description: {code:java} df.select(udf(lambda x: x, returnType=ExamplePointUDT(), useArrow=useArrow)(

[jira] [Created] (SPARK-48667) Arrow python UDFS didn't support UDT as output type

2024-06-19 Thread angerszhu (Jira)
angerszhu created SPARK-48667: - Summary: Arrow python UDFS didn't support UDT as output type Key: SPARK-48667 URL: https://issues.apache.org/jira/browse/SPARK-48667 Project: Spark Issue Type: Sub

[jira] [Resolved] (SPARK-48479) Support creating SQL functions in parser

2024-06-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48479. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46816 [https://gith

[jira] [Assigned] (SPARK-48479) Support creating SQL functions in parser

2024-06-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48479: --- Assignee: Allison Wang > Support creating SQL functions in parser > ---

[jira] [Updated] (SPARK-48666) A filter should not be pushed down if it contains Unevaluable expression

2024-06-19 Thread Wei Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated SPARK-48666: -- Description: We should avoid pushing down Unevaluable expression as it can cause unexpected failures.

[jira] [Updated] (SPARK-48666) A filter should not be pushed down if it contains Unevaluable expression

2024-06-19 Thread Wei Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated SPARK-48666: -- Description: We should avoid pushing down Unevaluable expression as it can cause unexpected failures.

[jira] [Created] (SPARK-48666) A filter should not be pushed down if it contains Unevaluable expression

2024-06-19 Thread Wei Zheng (Jira)
Wei Zheng created SPARK-48666: - Summary: A filter should not be pushed down if it contains Unevaluable expression Key: SPARK-48666 URL: https://issues.apache.org/jira/browse/SPARK-48666 Project: Spark

[jira] [Commented] (SPARK-48423) Unable to write MLPipeline to blob storage using .option attribute

2024-06-19 Thread Chhavi Bansal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856332#comment-17856332 ] Chhavi Bansal commented on SPARK-48423: --- [~weichenxu123]  sorry for tagging you, b

[jira] [Resolved] (SPARK-48541) Assign specific exit code for executors killed by TaskReaper

2024-06-19 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-48541. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46883 [https://github

[jira] [Created] (SPARK-48665) Support dict type for pyspark.sql.functions.lit

2024-06-19 Thread Ron Serruya (Jira)
Ron Serruya created SPARK-48665: --- Summary: Support dict type for pyspark.sql.functions.lit Key: SPARK-48665 URL: https://issues.apache.org/jira/browse/SPARK-48665 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-46957) Migrated shuffle data files from the decommissioned node should be removed when job completed

2024-06-19 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856284#comment-17856284 ] wuyi commented on SPARK-46957: -- [~yujhe.li] Thanks for reporting this bug. I'll submit a fi

[jira] [Updated] (SPARK-48664) Add official image Dockerfile for Apache Spark 4.0.0-preview1

2024-06-19 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-48664: - Summary: Add official image Dockerfile for Apache Spark 4.0.0-preview1 (was: Add official image Dockerf

[jira] [Created] (SPARK-48664) Add official image Dockerfile for Apache Spark 3.5.1

2024-06-19 Thread Kent Yao (Jira)
Kent Yao created SPARK-48664: Summary: Add official image Dockerfile for Apache Spark 3.5.1 Key: SPARK-48664 URL: https://issues.apache.org/jira/browse/SPARK-48664 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-48663) Use a double-check locking mechanism to minimize synchronization overhead in NettyUtils

2024-06-19 Thread Ricky Ma (Jira)
Ricky Ma created SPARK-48663: Summary: Use a double-check locking mechanism to minimize synchronization overhead in NettyUtils Key: SPARK-48663 URL: https://issues.apache.org/jira/browse/SPARK-48663 Proje

[jira] [Assigned] (SPARK-48656) ArrayIndexOutOfBoundsException in CartesianRDD getPartitions

2024-06-19 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48656: -- Assignee: Apache Spark > ArrayIndexOutOfBoundsException in CartesianRDD getPartitions

[jira] [Assigned] (SPARK-48656) ArrayIndexOutOfBoundsException in CartesianRDD getPartitions

2024-06-19 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48656: -- Assignee: (was: Apache Spark) > ArrayIndexOutOfBoundsException in CartesianRDD ge

[jira] [Commented] (SPARK-48656) ArrayIndexOutOfBoundsException in CartesianRDD getPartitions

2024-06-19 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856226#comment-17856226 ] ASF GitHub Bot commented on SPARK-48656: User 'wayneguow' has created a pull req

[jira] [Assigned] (SPARK-48578) Add new expressions for UTF8 string validation

2024-06-19 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48578: -- Assignee: Apache Spark > Add new expressions for UTF8 string validation > ---

[jira] [Assigned] (SPARK-48578) Add new expressions for UTF8 string validation

2024-06-19 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48578: -- Assignee: (was: Apache Spark) > Add new expressions for UTF8 string validation >

[jira] [Created] (SPARK-48662) Fix StructsToXml expression with collations

2024-06-19 Thread Mihailo Milosevic (Jira)
Mihailo Milosevic created SPARK-48662: - Summary: Fix StructsToXml expression with collations Key: SPARK-48662 URL: https://issues.apache.org/jira/browse/SPARK-48662 Project: Spark Issue T

[jira] [Assigned] (SPARK-48472) Enable reflect expressions with collated strings

2024-06-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48472: --- Assignee: Mihailo Milosevic > Enable reflect expressions with collated strings > --

[jira] [Resolved] (SPARK-48472) Enable reflect expressions with collated strings

2024-06-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48472. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46991 [https://gith

[jira] [Assigned] (SPARK-48574) Fix support for StructTypes with collations

2024-06-19 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48574: Assignee: Mihailo Milosevic > Fix support for StructTypes with collations > -

[jira] [Resolved] (SPARK-48574) Fix support for StructTypes with collations

2024-06-19 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48574. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46997 [https://github.com

[jira] [Resolved] (SPARK-48466) Wrap empty relation propagation in a dedicated node

2024-06-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48466. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46830 [https://gith