[jira] [Resolved] (SPARK-45513) Replace `scala.runtime.Tuple2Zipped` to `scala.collection.LazyZip2`

2023-10-13 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng resolved SPARK-45513. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43351 [https://github

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44662: --- Labels: pull-request-available (was: ) > SPIP: Improving performance of BroadcastHashJoin q

[jira] [Updated] (SPARK-45534) Use `java.lang.ref.Cleaner` instead of `finalize` for `RemoteBlockPushResolver`

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45534: --- Labels: pull-request-available (was: ) > Use `java.lang.ref.Cleaner` instead of `finalize`

[jira] [Comment Edited] (SPARK-45539) Add assert and log error if watermark predicate is missing in append mode and streaming aggregation

2023-10-13 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775103#comment-17775103 ] Anish Shrigondekar edited comment on SPARK-45539 at 10/14/23 1:24 AM:

[jira] [Resolved] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-10-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-44735. -- Fix Version/s: 4.0.0 Resolution: Fixed > Log a warning when inserting columns with the

[jira] [Assigned] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-10-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-44735: Assignee: Jia Fan > Log a warning when inserting columns with the same name by row that d

[jira] [Created] (SPARK-45539) Add assert and log error if watermark predicate is missing in append mode and streaming aggregation

2023-10-13 Thread Anish Shrigondekar (Jira)
Anish Shrigondekar created SPARK-45539: -- Summary: Add assert and log error if watermark predicate is missing in append mode and streaming aggregation Key: SPARK-45539 URL: https://issues.apache.org/jira/brows

[jira] [Updated] (SPARK-44210) Strengthen type checking and better comply with Connect specifications for `levenshtein` function

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44210: --- Labels: pull-request-available (was: ) > Strengthen type checking and better comply with Co

[jira] [Updated] (SPARK-43879) Decouple handle command and send response on server side

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43879: --- Labels: pull-request-available (was: ) > Decouple handle command and send response on serve

[jira] [Resolved] (SPARK-45487) Replace: _LEGACY_ERROR_TEMP_3007

2023-10-13 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-45487. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43342 [https:

[jira] [Assigned] (SPARK-45487) Replace: _LEGACY_ERROR_TEMP_3007

2023-10-13 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-45487: -- Assignee: Serge Rielau > Replace: _LEGACY_ERROR_TEMP_3007 > -

[jira] [Updated] (SPARK-45517) Expand more exception constructors to support error framework parameters

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45517: --- Labels: pull-request-available (was: ) > Expand more exception constructors to support erro

[jira] [Updated] (SPARK-45508) Add "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" so Platform can access cleaner on Java 9+

2023-10-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45508: -- Affects Version/s: 3.4.1 3.3.2 > Add "--add-opens=java.base/jdk.interna

[jira] [Updated] (SPARK-45508) Add "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" so Platform can access cleaner on Java 9+

2023-10-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45508: -- Fix Version/s: 3.4.2 3.5.1 3.3.4 > Add "--add-opens=java

[jira] [Assigned] (SPARK-45426) [CORE] Add support for a ReloadingTrustManager

2023-10-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45426: --- Assignee: Hasnain Lakhani > [CORE] Add support for a ReloadingTrustManager

[jira] [Resolved] (SPARK-45426) [CORE] Add support for a ReloadingTrustManager

2023-10-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45426. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 432

[jira] [Comment Edited] (SPARK-44679) java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2023-10-13 Thread huangsheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774963#comment-17774963 ] huangsheng edited comment on SPARK-44679 at 10/13/23 4:12 PM:

[jira] [Comment Edited] (SPARK-44679) java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2023-10-13 Thread huangsheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774963#comment-17774963 ] huangsheng edited comment on SPARK-44679 at 10/13/23 4:11 PM:

[jira] [Comment Edited] (SPARK-44679) java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2023-10-13 Thread huangsheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774963#comment-17774963 ] huangsheng edited comment on SPARK-44679 at 10/13/23 4:07 PM:

[jira] [Comment Edited] (SPARK-44679) java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2023-10-13 Thread huangsheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774963#comment-17774963 ] huangsheng edited comment on SPARK-44679 at 10/13/23 4:00 PM:

[jira] [Comment Edited] (SPARK-44679) java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2023-10-13 Thread huangsheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774963#comment-17774963 ] huangsheng edited comment on SPARK-44679 at 10/13/23 3:57 PM:

[jira] [Resolved] (SPARK-45495) Support stage level task resource profile for k8s cluster when dynamic allocation disabled

2023-10-13 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-45495. --- Resolution: Fixed > Support stage level task resource profile for k8s cluster when dynamic

[jira] [Commented] (SPARK-44679) java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2023-10-13 Thread huangsheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774963#comment-17774963 ] huangsheng commented on SPARK-44679: I think your problem is the same as [#SPARK-406

[jira] [Updated] (SPARK-45387) Partition key filter cannot be pushed down when using cast

2023-10-13 Thread TianyiMa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-45387: - Attachment: PruneFileSourcePartitions.diff > Partition key filter cannot be pushed down when using cast

[jira] [Commented] (SPARK-45527) Task fraction resource request is not expected

2023-10-13 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774957#comment-17774957 ] Thomas Graves commented on SPARK-45527: --- thanks for filing and digging into this.

[jira] [Assigned] (SPARK-45536) Lower the default -Xmx of build/mvn

2023-10-13 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45536: Assignee: Yang Jie > Lower the default -Xmx of build/mvn > --- >

[jira] [Resolved] (SPARK-45536) Lower the default -Xmx of build/mvn

2023-10-13 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45536. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43364 [https://github.com

[jira] [Updated] (SPARK-45538) pyspark connect overwrite_partitions bug

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45538: --- Labels: patch pull-request-available (was: patch) > pyspark connect overwrite_partitions bu

[jira] [Assigned] (SPARK-45476) Raise exception directly instead of calling `resolveColumnsByPosition`

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45476: -- Assignee: (was: Apache Spark) > Raise exception directly instead of calling `reso

[jira] [Assigned] (SPARK-45476) Raise exception directly instead of calling `resolveColumnsByPosition`

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45476: -- Assignee: Apache Spark > Raise exception directly instead of calling `resolveColumnsB

[jira] [Created] (SPARK-45538) pyspark connect overwrite_partitions bug

2023-10-13 Thread xie shuiahu (Jira)
xie shuiahu created SPARK-45538: --- Summary: pyspark connect overwrite_partitions bug Key: SPARK-45538 URL: https://issues.apache.org/jira/browse/SPARK-45538 Project: Spark Issue Type: Improvemen

[jira] [Resolved] (SPARK-45535) Do not compile tests for snapshots and release build

2023-10-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45535. -- Resolution: Invalid > Do not compile tests for snapshots and release build > -

[jira] [Updated] (SPARK-45537) The last Task may get stuck in multi resource-profile

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45537: --- Labels: pull-request-available (was: ) > The last Task may get stuck in multi resource-prof

[jira] [Created] (SPARK-45537) The last Task may get stuck in multi resource-profile

2023-10-13 Thread liangyongyuan (Jira)
liangyongyuan created SPARK-45537: - Summary: The last Task may get stuck in multi resource-profile Key: SPARK-45537 URL: https://issues.apache.org/jira/browse/SPARK-45537 Project: Spark Issue

[jira] [Updated] (SPARK-45536) Lower the default -Xmx of build/mvn

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45536: --- Labels: pull-request-available (was: ) > Lower the default -Xmx of build/mvn >

[jira] [Updated] (SPARK-45535) Do not compile tests for snapshots and release build

2023-10-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45535: --- Labels: pull-request-available (was: ) > Do not compile tests for snapshots and release bui

[jira] [Created] (SPARK-45536) Lower the default -Xmx of build/mvn

2023-10-13 Thread Yang Jie (Jira)
Yang Jie created SPARK-45536: Summary: Lower the default -Xmx of build/mvn Key: SPARK-45536 URL: https://issues.apache.org/jira/browse/SPARK-45536 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-45535) Do not compile tests for snapshots and release build

2023-10-13 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-45535: Summary: Do not compile tests for snapshots and release build Key: SPARK-45535 URL: https://issues.apache.org/jira/browse/SPARK-45535 Project: Spark Issue Ty