[jira] [Created] (SPARK-38681) Support nested generic case classes

2022-03-29 Thread Emil Ejbyfeldt (Jira)
Emil Ejbyfeldt created SPARK-38681: -- Summary: Support nested generic case classes Key: SPARK-38681 URL: https://issues.apache.org/jira/browse/SPARK-38681 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-38682) Complex calculations with lead to driver oom

2022-03-29 Thread JacobZheng (Jira)
JacobZheng created SPARK-38682: -- Summary: Complex calculations with lead to driver oom Key: SPARK-38682 URL: https://issues.apache.org/jira/browse/SPARK-38682 Project: Spark Issue Type: Improve

[jira] [Updated] (SPARK-38682) Complex calculations with lead to driver oom

2022-03-29 Thread JacobZheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-38682: --- Attachment: screenshot-1.png > Complex calculations with lead to driver oom > -

[jira] [Updated] (SPARK-38682) Complex calculations with lead to driver oom

2022-03-29 Thread JacobZheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-38682: --- Attachment: 20220329164645.jpg > Complex calculations with lead to driver oom > ---

[jira] [Updated] (SPARK-38682) Complex calculations with lead to driver oom

2022-03-29 Thread JacobZheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-38682: --- Description: My spark job is working fine in version 3.0.1. After I upgraded to 3.2, the driver wou

[jira] [Updated] (SPARK-38682) Complex calculations with lead to driver oom

2022-03-29 Thread JacobZheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-38682: --- Attachment: (was: 20220329164645.jpg) > Complex calculations with lead to driver oom >

[jira] [Created] (SPARK-38683) It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection

2022-03-29 Thread weixiuli (Jira)
weixiuli created SPARK-38683: Summary: It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated Key: SPARK-

[jira] [Created] (SPARK-38684) Stream-stream outer join has a possible correctness issue due to weakly read consistent on outer iterators

2022-03-29 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-38684: Summary: Stream-stream outer join has a possible correctness issue due to weakly read consistent on outer iterators Key: SPARK-38684 URL: https://issues.apache.org/jira/browse/SPA

[jira] [Commented] (SPARK-38684) Stream-stream outer join has a possible correctness issue due to weakly read consistent on outer iterators

2022-03-29 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513949#comment-17513949 ] Jungtaek Lim commented on SPARK-38684: -- Will submit a PR sooner. > Stream-stream o

[jira] [Commented] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513962#comment-17513962 ] Apache Spark commented on SPARK-38561: -- User 'Yikun' has created a pull request for

[jira] [Assigned] (SPARK-38683) It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38683: Assignee: Apache Spark > It is unnecessary to release the ShuffleManagedBufferIterator or

[jira] [Assigned] (SPARK-38683) It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38683: Assignee: (was: Apache Spark) > It is unnecessary to release the ShuffleManagedBuffer

[jira] [Commented] (SPARK-38683) It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connectio

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513963#comment-17513963 ] Apache Spark commented on SPARK-38683: -- User 'weixiuli' has created a pull request

[jira] [Assigned] (SPARK-38684) Stream-stream outer join has a possible correctness issue due to weakly read consistent on outer iterators

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38684: Assignee: (was: Apache Spark) > Stream-stream outer join has a possible correctness i

[jira] [Commented] (SPARK-38684) Stream-stream outer join has a possible correctness issue due to weakly read consistent on outer iterators

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513994#comment-17513994 ] Apache Spark commented on SPARK-38684: -- User 'HeartSaVioR' has created a pull reque

[jira] [Assigned] (SPARK-38684) Stream-stream outer join has a possible correctness issue due to weakly read consistent on outer iterators

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38684: Assignee: Apache Spark > Stream-stream outer join has a possible correctness issue due to

[jira] [Updated] (SPARK-38682) Complex calculations with lead to driver oom

2022-03-29 Thread JacobZheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-38682: --- Description: My spark job is working fine in version 3.0.1. After I upgraded to 3.2, the driver wou

[jira] [Resolved] (SPARK-38670) Add offset commit time to streaming query listener

2022-03-29 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-38670. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35985 [https://gi

[jira] [Assigned] (SPARK-38670) Add offset commit time to streaming query listener

2022-03-29 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-38670: Assignee: Boyang Jerry Peng > Add offset commit time to streaming query listener > --

[jira] [Created] (SPARK-38685) Improve the implement of percentile_cont

2022-03-29 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-38685: -- Summary: Improve the implement of percentile_cont Key: SPARK-38685 URL: https://issues.apache.org/jira/browse/SPARK-38685 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-38685) Improve the implement of percentile_cont

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38685: Assignee: Apache Spark > Improve the implement of percentile_cont > -

[jira] [Assigned] (SPARK-38685) Improve the implement of percentile_cont

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38685: Assignee: (was: Apache Spark) > Improve the implement of percentile_cont > --

[jira] [Commented] (SPARK-38685) Improve the implement of percentile_cont

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514062#comment-17514062 ] Apache Spark commented on SPARK-38685: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-38685) Improve the implement of percentile_cont

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514060#comment-17514060 ] Apache Spark commented on SPARK-38685: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-38681) Support nested generic case classes

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514064#comment-17514064 ] Apache Spark commented on SPARK-38681: -- User 'eejbyfeldt' has created a pull reques

[jira] [Assigned] (SPARK-38681) Support nested generic case classes

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38681: Assignee: (was: Apache Spark) > Support nested generic case classes > ---

[jira] [Assigned] (SPARK-38681) Support nested generic case classes

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38681: Assignee: Apache Spark > Support nested generic case classes > --

[jira] [Resolved] (SPARK-38674) Remove useless deduplicate in SubqueryBroadcastExec

2022-03-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38674. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35989 [https://gith

[jira] [Assigned] (SPARK-38674) Remove useless deduplicate in SubqueryBroadcastExec

2022-03-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38674: --- Assignee: Yuming Wang > Remove useless deduplicate in SubqueryBroadcastExec > -

[jira] [Assigned] (SPARK-38506) Push partial aggregation through join

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38506: Assignee: Apache Spark > Push partial aggregation through join >

[jira] [Assigned] (SPARK-38506) Push partial aggregation through join

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38506: Assignee: (was: Apache Spark) > Push partial aggregation through join > -

[jira] [Commented] (SPARK-38506) Push partial aggregation through join

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514133#comment-17514133 ] Apache Spark commented on SPARK-38506: -- User 'wangyum' has created a pull request f

[jira] [Resolved] (SPARK-38562) Add doc for Volcano scheduler

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38562. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35870 [https://

[jira] [Assigned] (SPARK-38562) Add doc for Volcano scheduler

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38562: - Assignee: Yikun Jiang > Add doc for Volcano scheduler > - >

[jira] [Updated] (SPARK-38562) Add doc for Volcano scheduler

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38562: -- Fix Version/s: 3.3.0 (was: 3.4.0) > Add doc for Volcano scheduler > ---

[jira] [Created] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

2022-03-29 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38686: Summary: Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates` Key: SPARK-38686 URL: https://issues.apache.org/jira/browse/SPARK-38686 Project: Spark

[jira] [Resolved] (SPARK-37982) Use error classes in the execution errors related to unsupported input type

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37982. -- Fix Version/s: 3.4.0 (was: 3.3.0) Resolution: Fixed Issue resolved by pu

[jira] [Assigned] (SPARK-37982) Use error classes in the execution errors related to unsupported input type

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-37982: Assignee: leesf > Use error classes in the execution errors related to unsupported input type > -

[jira] [Commented] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514275#comment-17514275 ] Apache Spark commented on SPARK-38686: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38686: Assignee: (was: Apache Spark) > Implement `keep` parameter of `(Index/MultiIndex).dro

[jira] [Assigned] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38686: Assignee: Apache Spark > Implement `keep` parameter of `(Index/MultiIndex).drop_duplicate

[jira] [Updated] (SPARK-38687) Use error classes in the compilation errors of generators

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38687: - Affects Version/s: 3.4.0 (was: 3.3.0) > Use error classes in the compilation

[jira] [Created] (SPARK-38687) Use error classes in the compilation errors of generators

2022-03-29 Thread Max Gekk (Jira)
Max Gekk created SPARK-38687: Summary: Use error classes in the compilation errors of generators Key: SPARK-38687 URL: https://issues.apache.org/jira/browse/SPARK-38687 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38687) Use error classes in the compilation errors of generators

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38687: - Description: Migrate the following errors in QueryCompilationErrors: * nestedGeneratorError * moreThanOn

[jira] [Updated] (SPARK-38688) Use error classes in the compilation errors of deserializer

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38688: - Description: Migrate the following errors in QueryCompilationErrors: * dataTypeMismatchForDeserializerEr

[jira] [Created] (SPARK-38688) Use error classes in the compilation errors of deserializer

2022-03-29 Thread Max Gekk (Jira)
Max Gekk created SPARK-38688: Summary: Use error classes in the compilation errors of deserializer Key: SPARK-38688 URL: https://issues.apache.org/jira/browse/SPARK-38688 Project: Spark Issue Ty

[jira] [Updated] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38689: - Affects Version/s: 3.4.0 (was: 3.3.0) > Use error classes in the compilation

[jira] [Created] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-03-29 Thread Max Gekk (Jira)
Max Gekk created SPARK-38689: Summary: Use error classes in the compilation errors of not allowed DESC PARTITION Key: SPARK-38689 URL: https://issues.apache.org/jira/browse/SPARK-38689 Project: Spark

[jira] [Updated] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38689: - Description: Migrate the following errors in QueryCompilationErrors: * descPartitionNotAllowedOnTempView

[jira] [Updated] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38689: - Description: Migrate the following errors in QueryCompilationErrors: * descPartitionNotAllowedOnTempView

[jira] [Created] (SPARK-38690) Use error classes in the compilation errors of SHOW CREATE TABLE

2022-03-29 Thread Max Gekk (Jira)
Max Gekk created SPARK-38690: Summary: Use error classes in the compilation errors of SHOW CREATE TABLE Key: SPARK-38690 URL: https://issues.apache.org/jira/browse/SPARK-38690 Project: Spark Iss

[jira] [Updated] (SPARK-38690) Use error classes in the compilation errors of SHOW CREATE TABLE

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38690: - Description: Migrate the following errors in QueryCompilationErrors: * showCreateTableAsSerdeNotSupporte

[jira] [Created] (SPARK-38691) Use error classes in the compilation errors of column/attr resolving

2022-03-29 Thread Max Gekk (Jira)
Max Gekk created SPARK-38691: Summary: Use error classes in the compilation errors of column/attr resolving Key: SPARK-38691 URL: https://issues.apache.org/jira/browse/SPARK-38691 Project: Spark

[jira] [Updated] (SPARK-38691) Use error classes in the compilation errors of column/attr resolving

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38691: - Description: Migrate the following errors in QueryCompilationErrors: * cannotResolveUserSpecifiedColumns

[jira] [Created] (SPARK-38692) Use error classes in the compilation errors of function args

2022-03-29 Thread Max Gekk (Jira)
Max Gekk created SPARK-38692: Summary: Use error classes in the compilation errors of function args Key: SPARK-38692 URL: https://issues.apache.org/jira/browse/SPARK-38692 Project: Spark Issue T

[jira] [Updated] (SPARK-38692) Use error classes in the compilation errors of function args

2022-03-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38692: - Description: Migrate the following errors in QueryCompilationErrors: * invalidFunctionArgumentsError * i

[jira] [Created] (SPARK-38693) Spark does not use SessionManager

2022-03-29 Thread Brad Solomon (Jira)
Brad Solomon created SPARK-38693: Summary: Spark does not use SessionManager Key: SPARK-38693 URL: https://issues.apache.org/jira/browse/SPARK-38693 Project: Spark Issue Type: Bug C

[jira] [Updated] (SPARK-38693) Spark does not use SessionManager

2022-03-29 Thread Brad Solomon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad Solomon updated SPARK-38693: - Description: Spark's failure to use a `SessionManager` causes `java.lang.IllegalStateException:

[jira] [Updated] (SPARK-38693) Spark does not use SessionManager

2022-03-29 Thread Brad Solomon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad Solomon updated SPARK-38693: - Description: Spark's failure to use a `SessionManager` causes `java.lang.IllegalStateException:

[jira] [Updated] (SPARK-38693) Spark does not use SessionManager

2022-03-29 Thread Brad Solomon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad Solomon updated SPARK-38693: - Description: Spark's failure to use a `SessionManager` causes `java.lang.IllegalStateException:

[jira] [Updated] (SPARK-35803) Spark SQL does not support creating views using DataSource v2 based data sources

2022-03-29 Thread David Rabinowitz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Rabinowitz updated SPARK-35803: - Issue Type: Bug (was: New Feature) > Spark SQL does not support creating views using Da

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514343#comment-17514343 ] Dongjoon Hyun commented on SPARK-38652: --- Any update, [~dcoliversun]? > K8S IT Tes

[jira] [Commented] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514364#comment-17514364 ] Dongjoon Hyun commented on SPARK-33349: --- Hi, All. Could you try to use the latest

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514379#comment-17514379 ] Dongjoon Hyun commented on SPARK-38652: --- BTW, [~dcoliversun]. K8s IT itself doesn'

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread qian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514390#comment-17514390 ] qian commented on SPARK-38652: -- [~dongjoon] Hi. DepsTestsSuite has tests as follow * Launc

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514398#comment-17514398 ] Dongjoon Hyun commented on SPARK-38652: --- Got it, [~dcoliversun]. > K8S IT Test De

[jira] [Updated] (SPARK-38320) (flat)MapGroupsWithState can timeout groups which just received inputs in the same microbatch

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38320: -- Labels: correctness (was: ) > (flat)MapGroupsWithState can timeout groups which just received

[jira] [Created] (SPARK-38694) Simplify Java UT code with Junit `assertThrows`

2022-03-29 Thread Yang Jie (Jira)
Yang Jie created SPARK-38694: Summary: Simplify Java UT code with Junit `assertThrows` Key: SPARK-38694 URL: https://issues.apache.org/jira/browse/SPARK-38694 Project: Spark Issue Type: Improveme

[jira] [Commented] (SPARK-38694) Simplify Java UT code with Junit `assertThrows`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514412#comment-17514412 ] Apache Spark commented on SPARK-38694: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-38694) Simplify Java UT code with Junit `assertThrows`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38694: Assignee: Apache Spark > Simplify Java UT code with Junit `assertThrows` > --

[jira] [Commented] (SPARK-38694) Simplify Java UT code with Junit `assertThrows`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514413#comment-17514413 ] Apache Spark commented on SPARK-38694: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-38694) Simplify Java UT code with Junit `assertThrows`

2022-03-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38694: Assignee: (was: Apache Spark) > Simplify Java UT code with Junit `assertThrows` > ---

[jira] [Assigned] (SPARK-38349) No need to filter events when sessionwindow gapDuration greater than 0

2022-03-29 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-38349: Assignee: nyingping > No need to filter events when sessionwindow gapDuration greater tha

[jira] [Resolved] (SPARK-38349) No need to filter events when sessionwindow gapDuration greater than 0

2022-03-29 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-38349. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35680 [https://gi

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514431#comment-17514431 ] Dongjoon Hyun commented on SPARK-38652: --- I also confirmed this regression and rais

[jira] [Updated] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38652: -- Priority: Blocker (was: Major) > K8S IT Test DepsTestsSuite blocks with PathIOException in ha

[jira] [Updated] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38652: -- Component/s: (was: Tests) > K8S IT Test DepsTestsSuite blocks with PathIOException in hado

[jira] [Resolved] (SPARK-38605) Retrying on file manager operation in HDFSMetadataLog

2022-03-29 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-38605. - Resolution: Won't Fix > Retrying on file manager operation in HDFSMetadataLog >