[jira] [Resolved] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38051. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35349 [https://

[jira] [Assigned] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38051: - Assignee: Dongjoon Hyun > Update Roxygen reference to 7.1.2 > -

[jira] [Commented] (SPARK-37994) Unable to build spark3.2 with -Dhadoop.version=3.1.4

[ https://issues.apache.org/jira/browse/SPARK-37994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483605#comment-17483605 ] Vu Tan commented on SPARK-37994: [~csun]  Thank you for your kindness!  I understood t

[jira] [Commented] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483604#comment-17483604 ] XiDuo You commented on SPARK-33832: --- thank you [~dongjoon] ! > Add an option in AQE t

[jira] [Updated] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-38045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-38045: - Fix Version/s: 3.2.2 > More strict validation on plan check for stream-stream join unit test > -

[jira] [Commented] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483601#comment-17483601 ] Apache Spark commented on SPARK-38013: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483600#comment-17483600 ] Apache Spark commented on SPARK-38013: -- User 'ulysses-you' has created a pull reque

[jira] [Updated] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Task) > AQE can change bhj to smj if no extra

[jira] [Commented] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483598#comment-17483598 ] XiDuo You commented on SPARK-38013: --- Add a test to cover this behavior > AQE can chan

[jira] [Commented] (SPARK-37931) Quote the column name if needed

[ https://issues.apache.org/jira/browse/SPARK-37931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483589#comment-17483589 ] Apache Spark commented on SPARK-37931: -- User 'Peng-Lei' has created a pull request

[jira] [Commented] (SPARK-38052) Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable

[ https://issues.apache.org/jira/browse/SPARK-38052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483585#comment-17483585 ] Apache Spark commented on SPARK-38052: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-38052) Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable

[ https://issues.apache.org/jira/browse/SPARK-38052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38052: Assignee: (was: Apache Spark) > Use loops to refactor methods UnsafeRow#isFixedLength

[jira] [Assigned] (SPARK-38052) Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable

[ https://issues.apache.org/jira/browse/SPARK-38052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38052: Assignee: Apache Spark > Use loops to refactor methods UnsafeRow#isFixedLength and Unsafe

[jira] [Commented] (SPARK-38052) Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable

[ https://issues.apache.org/jira/browse/SPARK-38052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483584#comment-17483584 ] Apache Spark commented on SPARK-38052: -- User 'LuciferYang' has created a pull reque

[jira] [Updated] (SPARK-38052) Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable

[ https://issues.apache.org/jira/browse/SPARK-38052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-38052: - Description: Methods UnsafeRow#isFixedLength and UnsafeRow#isMutable use tail recursion now , we can rew

[jira] [Commented] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483581#comment-17483581 ] Apache Spark commented on SPARK-38051: -- User 'dongjoon-hyun' has created a pull req

[jira] [Updated] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38051: -- Summary: Update Roxygen reference to 7.1.2 (was: Upgrade Roxygen to 7.1.2) > Update Roxygen r

[jira] [Commented] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483579#comment-17483579 ] Apache Spark commented on SPARK-38051: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38051: Assignee: (was: Apache Spark) > Update Roxygen reference to 7.1.2 > -

[jira] [Assigned] (SPARK-38051) Update Roxygen reference to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38051: Assignee: Apache Spark > Update Roxygen reference to 7.1.2 >

[jira] [Created] (SPARK-38052) Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable

Yang Jie created SPARK-38052: Summary: Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable Key: SPARK-38052 URL: https://issues.apache.org/jira/browse/SPARK-38052 Project: Spark

[jira] [Updated] (SPARK-38051) Upgrade Roxygen to 7.1.2

[ https://issues.apache.org/jira/browse/SPARK-38051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38051: -- Component/s: Documentation R (was: Build) > Upgrade Roxy

[jira] [Created] (SPARK-38051) Upgrade Roxygen to 7.1.2

Dongjoon Hyun created SPARK-38051: - Summary: Upgrade Roxygen to 7.1.2 Key: SPARK-38051 URL: https://issues.apache.org/jira/browse/SPARK-38051 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-38035) Add docker tests for build-in JDBC dialect

[ https://issues.apache.org/jira/browse/SPARK-38035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483568#comment-17483568 ] Apache Spark commented on SPARK-38035: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-38045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483565#comment-17483565 ] Apache Spark commented on SPARK-38045: -- User 'HeartSaVioR' has created a pull reque

[jira] [Created] (SPARK-38050) Spark Job fails intermittently as the connection times out while checking the driver pod's status during the creation of ExecutorPodsAllocator.

Ankit Prakash Gupta created SPARK-38050: --- Summary: Spark Job fails intermittently as the connection times out while checking the driver pod's status during the creation of ExecutorPodsAllocator. Key: SPARK-38050

[jira] [Commented] (SPARK-38049) Use Java 17 in K8s integration tests

[ https://issues.apache.org/jira/browse/SPARK-38049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483562#comment-17483562 ] Apache Spark commented on SPARK-38049: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-38049) Use Java 17 in K8s integration tests

[ https://issues.apache.org/jira/browse/SPARK-38049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38049: - Assignee: Dongjoon Hyun > Use Java 17 in K8s integration tests > --

[jira] [Created] (SPARK-38049) Use Java 17 in K8s integration tests

Dongjoon Hyun created SPARK-38049: - Summary: Use Java 17 in K8s integration tests Key: SPARK-38049 URL: https://issues.apache.org/jira/browse/SPARK-38049 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-37145) Improvement for extending pod feature steps with KubernetesConf

[ https://issues.apache.org/jira/browse/SPARK-37145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483540#comment-17483540 ] Apache Spark commented on SPARK-37145: -- User 'Yikun' has created a pull request for

[jira] [Commented] (SPARK-28001) Dataframe throws 'socket.timeout: timed out' exception

[ https://issues.apache.org/jira/browse/SPARK-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483539#comment-17483539 ] Mr.OverDose commented on SPARK-28001: - I have the exact same issue here, although it

[jira] [Commented] (SPARK-36872) Decommissioning executors get killed before transferring their data because of the hardcoded timeout of 60 secs

[ https://issues.apache.org/jira/browse/SPARK-36872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483537#comment-17483537 ] Abhishek Rao commented on SPARK-36872: -- Thanks. We'll have a look at this. > Decom

[jira] [Assigned] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38048: - Assignee: William Hyun > Add IntegrationTestBackend.describePods to support all K8s tes

[jira] [Resolved] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38048. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35344 [https://

[jira] [Commented] (SPARK-38041) DataFilter pushed down with PartitionFilter

[ https://issues.apache.org/jira/browse/SPARK-38041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483527#comment-17483527 ] Jackey Lee commented on SPARK-38041: Thanks for your attention, [~hyukjin.kwon]. I'm

[jira] [Commented] (SPARK-38041) DataFilter pushed down with PartitionFilter

[ https://issues.apache.org/jira/browse/SPARK-38041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483518#comment-17483518 ] Hyukjin Kwon commented on SPARK-38041: -- [~Jackey Lee] mind showing a self-contained

[jira] [Updated] (SPARK-38041) DataFilter pushed down with PartitionFilter

[ https://issues.apache.org/jira/browse/SPARK-38041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38041: - Description: At present, the Filter is divided into DataFilter and PartitionFilter when it is p

[jira] [Commented] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483517#comment-17483517 ] Apache Spark commented on SPARK-38048: -- User 'williamhyun' has created a pull reque

[jira] [Assigned] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38048: Assignee: Apache Spark > Add IntegrationTestBackend.describePods to support all K8s test

[jira] [Assigned] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38048: Assignee: (was: Apache Spark) > Add IntegrationTestBackend.describePods to support al

[jira] [Created] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

William Hyun created SPARK-38048: Summary: Add IntegrationTestBackend.describePods to support all K8s test backends Key: SPARK-38048 URL: https://issues.apache.org/jira/browse/SPARK-38048 Project: Spa

[jira] [Updated] (SPARK-36057) Support Customized Kubernetes Schedulers

[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated SPARK-36057: Summary: Support Customized Kubernetes Schedulers (was: Support volcano/alternative schedulers)

[jira] [Commented] (SPARK-36060) Support backing off dynamic allocation increases if resources are "stuck"

[ https://issues.apache.org/jira/browse/SPARK-36060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483504#comment-17483504 ] Weiwei Yang commented on SPARK-36060: - hi [~holden]  We have seen the same issue. T

[jira] [Updated] (SPARK-38042) Encoder cannot be found when a tuple component is a type alias for an Array

[ https://issues.apache.org/jira/browse/SPARK-38042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Nyström-Persson updated SPARK-38042: -- Description: ScalaReflection.dataTypeFor fails when Array[T] has been aliased

[jira] [Assigned] (SPARK-38047) Provide an option to only roll executors if they are outliers

[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38047: - Assignee: Alex Holmes > Provide an option to only roll executors if they are outliers >

[jira] [Updated] (SPARK-38047) Provide an option to only roll executors if they are outliers

[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38047: -- Fix Version/s: (was: 3.3.0) > Provide an option to only roll executors if they are outlier

[jira] [Updated] (SPARK-38047) Provide an option to only roll executors if they are outliers

[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Holmes updated SPARK-38047: Description: Currently executor rolling will always kill one executor every {{{}spark.kubernetes.

[jira] [Created] (SPARK-38047) Provide an option to only roll executors if they are outliers

Alex Holmes created SPARK-38047: --- Summary: Provide an option to only roll executors if they are outliers Key: SPARK-38047 URL: https://issues.apache.org/jira/browse/SPARK-38047 Project: Spark

[jira] [Updated] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

[ https://issues.apache.org/jira/browse/SPARK-38046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-38046: -- Description: There is a test call "compositeReadLimit"   [https://github.com/apache/

[jira] [Updated] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

[ https://issues.apache.org/jira/browse/SPARK-38046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-38046: -- Description: There is a test call "compositeReadLimit"   [https://github.com/apache/

[jira] [Assigned] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

[ https://issues.apache.org/jira/browse/SPARK-38046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38046: Assignee: (was: Apache Spark) > Fix KafkaSource/KafkaMicroBatch flaky test due to non

[jira] [Commented] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

[ https://issues.apache.org/jira/browse/SPARK-38046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483432#comment-17483432 ] Apache Spark commented on SPARK-38046: -- User 'jerrypeng' has created a pull request

[jira] [Assigned] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

[ https://issues.apache.org/jira/browse/SPARK-38046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38046: Assignee: Apache Spark > Fix KafkaSource/KafkaMicroBatch flaky test due to non-determinis

[jira] [Resolved] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-38045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38045. --- Fix Version/s: 3.3.0 Assignee: Jungtaek Lim Resolution: Fixed This is resolv

[jira] [Created] (SPARK-38046) Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing

Boyang Jerry Peng created SPARK-38046: - Summary: Fix KafkaSource/KafkaMicroBatch flaky test due to non-deterministic timing Key: SPARK-38046 URL: https://issues.apache.org/jira/browse/SPARK-38046

[jira] [Closed] (SPARK-33828) SQL Adaptive Query Execution QA

[ https://issues.apache.org/jira/browse/SPARK-33828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-33828. - > SQL Adaptive Query Execution QA > --- > > Key: SPARK-3

[jira] [Updated] (SPARK-36652) AQE dynamic join selection should not apply to non-equi join

[ https://issues.apache.org/jira/browse/SPARK-36652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36652: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Bug) > AQE dynamic join selection shou

[jira] [Updated] (SPARK-36652) AQE dynamic join selection should not apply to non-equi join

[ https://issues.apache.org/jira/browse/SPARK-36652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36652: -- Parent: (was: SPARK-33828) Issue Type: Bug (was: Sub-task) > AQE dynamic join sel

[jira] [Updated] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-37675: -- Description: We identified 3 issues in the handling of merge finalization requests in the Rem

[jira] [Updated] (SPARK-36424) Support eliminate limits in AQE Optimizer

[ https://issues.apache.org/jira/browse/SPARK-36424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36424: -- Parent: (was: SPARK-33828) Issue Type: Improvement (was: Sub-task) > Support elim

[jira] [Updated] (SPARK-34980) Support coalesce partition through union

[ https://issues.apache.org/jira/browse/SPARK-34980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34980: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Bug) > Support coalesce partition thro

[jira] [Updated] (SPARK-36424) Support eliminate limits in AQE Optimizer

[ https://issues.apache.org/jira/browse/SPARK-36424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36424: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Improvement) > Support eliminate limit

[jira] [Updated] (SPARK-34980) Support coalesce partition through union

[ https://issues.apache.org/jira/browse/SPARK-34980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34980: -- Parent: (was: SPARK-33828) Issue Type: Bug (was: Sub-task) > Support coalesce par

[jira] [Updated] (SPARK-37063) SQL Adaptive Query Execution QA: Phase 2

[ https://issues.apache.org/jira/browse/SPARK-37063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37063: -- Target Version/s: 3.3.0 > SQL Adaptive Query Execution QA: Phase 2 > -

[jira] [Commented] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483411#comment-17483411 ] Dongjoon Hyun commented on SPARK-33832: --- I switched the umbrella issue to SPARK-37

[jira] [Updated] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33832: -- Parent: (was: SPARK-33828) Issue Type: Bug (was: Sub-task) > Add an option in AQE

[jira] [Updated] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33832: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Bug) > Add an option in AQE to mitigat

[jira] [Updated] (SPARK-37328) SPARK-33832 brings the bug that OptimizeSkewedJoin may not work since it was applied on whole plan innstead of new stage plan

[ https://issues.apache.org/jira/browse/SPARK-37328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37328: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Bug) > SPARK-33832 brings the bug that

[jira] [Commented] (SPARK-37193) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should not apply to outer joins

[ https://issues.apache.org/jira/browse/SPARK-37193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483408#comment-17483408 ] Dongjoon Hyun commented on SPARK-37193: --- I removed the `Fix Version` because this

[jira] [Updated] (SPARK-37193) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should not apply to outer joins

[ https://issues.apache.org/jira/browse/SPARK-37193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37193: -- Fix Version/s: (was: 3.3.0) > DynamicJoinSelection.shouldDemoteBroadcastHashJoin should no

[jira] [Comment Edited] (SPARK-37193) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should not apply to outer joins

[ https://issues.apache.org/jira/browse/SPARK-37193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483408#comment-17483408 ] Dongjoon Hyun edited comment on SPARK-37193 at 1/27/22, 7:57 PM: -

[jira] [Assigned] (SPARK-35442) Support propagate empty relation through aggregate

[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35442: - Assignee: XiDuo You > Support propagate empty relation through aggregate >

[jira] [Commented] (SPARK-37442) In AQE, wrong InMemoryRelation size estimation causes "Cannot broadcast the table that is larger than 8GB: 8 GB" failure

[ https://issues.apache.org/jira/browse/SPARK-37442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483405#comment-17483405 ] Dongjoon Hyun commented on SPARK-37442: --- I converted this into SPARK-37063's subta

[jira] [Updated] (SPARK-37442) In AQE, wrong InMemoryRelation size estimation causes "Cannot broadcast the table that is larger than 8GB: 8 GB" failure

[ https://issues.apache.org/jira/browse/SPARK-37442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37442: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Bug) > In AQE, wrong InMemoryRelation

[jira] [Assigned] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-38045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38045: Assignee: Apache Spark > More strict validation on plan check for stream-stream join unit

[jira] [Assigned] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-38045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38045: Assignee: (was: Apache Spark) > More strict validation on plan check for stream-strea

[jira] [Commented] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-38045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483403#comment-17483403 ] Apache Spark commented on SPARK-38045: -- User 'HeartSaVioR' has created a pull reque

[jira] [Commented] (SPARK-37994) Unable to build spark3.2 with -Dhadoop.version=3.1.4

[ https://issues.apache.org/jira/browse/SPARK-37994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483399#comment-17483399 ] Chao Sun commented on SPARK-37994: -- Glad it helped [~tanvu]! {quote} We can omit the -

[jira] [Updated] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-37675: -- Summary: Push-based merge finalization bugs in the RemoteBlockPushResolver (was: Return PushM

[jira] [Created] (SPARK-38045) More strict validation on plan check for stream-stream join unit test

Dongjoon Hyun created SPARK-38045: - Summary: More strict validation on plan check for stream-stream join unit test Key: SPARK-38045 URL: https://issues.apache.org/jira/browse/SPARK-38045 Project: Spar

[jira] [Assigned] (SPARK-35693) Add plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-35693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35693: - Assignee: Cheng Su (was: Apache Spark) > Add plan check for stream-stream join unit te

[jira] [Commented] (SPARK-36872) Decommissioning executors get killed before transferring their data because of the hardcoded timeout of 60 secs

[ https://issues.apache.org/jira/browse/SPARK-36872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483281#comment-17483281 ] Shekhar Gupta commented on SPARK-36872: --- [~abhisrao] I am referring to the followi

[jira] [Created] (SPARK-38044) Spark dynamic allocation ignores pending jobs

Igor Calabria created SPARK-38044: - Summary: Spark dynamic allocation ignores pending jobs Key: SPARK-38044 URL: https://issues.apache.org/jira/browse/SPARK-38044 Project: Spark Issue Type: B

[jira] [Updated] (SPARK-35216) a general auto merge output small files feature for datasource api

[ https://issues.apache.org/jira/browse/SPARK-35216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qian wang updated SPARK-35216: -- Description: in most case, users write data to hive table or hdfs dir with spark sql, since as spark3

[jira] [Commented] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483162#comment-17483162 ] Hamza Khribi commented on SPARK-19335: -- Hello [~kevinyu98] , is there any news abou

[jira] [Resolved] (SPARK-38001) Replace the unsupported error classes by `UNSUPPORTED_FEATURE`

[ https://issues.apache.org/jira/browse/SPARK-38001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38001. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35302 [https://gith

[jira] [Commented] (SPARK-38043) Refactor FileDataSourceBaseSuite

[ https://issues.apache.org/jira/browse/SPARK-38043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483126#comment-17483126 ] Apache Spark commented on SPARK-38043: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-38043) Refactor FileDataSourceBaseSuite

[ https://issues.apache.org/jira/browse/SPARK-38043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38043: Assignee: (was: Apache Spark) > Refactor FileDataSourceBaseSuite > --

[jira] [Assigned] (SPARK-38043) Refactor FileDataSourceBaseSuite

[ https://issues.apache.org/jira/browse/SPARK-38043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38043: Assignee: Apache Spark > Refactor FileDataSourceBaseSuite > -

[jira] [Created] (SPARK-38043) Refactor FileDataSourceBaseSuite

angerszhu created SPARK-38043: - Summary: Refactor FileDataSourceBaseSuite Key: SPARK-38043 URL: https://issues.apache.org/jira/browse/SPARK-38043 Project: Spark Issue Type: Task Compone

[jira] [Commented] (SPARK-35693) Add plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-35693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482987#comment-17482987 ] Apache Spark commented on SPARK-35693: -- User 'HeartSaVioR' has created a pull reque

[jira] [Commented] (SPARK-35693) Add plan check for stream-stream join unit test

[ https://issues.apache.org/jira/browse/SPARK-35693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482986#comment-17482986 ] Apache Spark commented on SPARK-35693: -- User 'HeartSaVioR' has created a pull reque

[jira] [Resolved] (SPARK-37983) Backout agg build time metrics from sort aggregate

[ https://issues.apache.org/jira/browse/SPARK-37983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37983. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35273 [https://gith

[jira] [Assigned] (SPARK-37983) Backout agg build time metrics from sort aggregate

[ https://issues.apache.org/jira/browse/SPARK-37983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37983: --- Assignee: Cheng Su > Backout agg build time metrics from sort aggregate > -

[jira] [Commented] (SPARK-38042) Encoder cannot be found when a tuple component is a type alias for an Array

[ https://issues.apache.org/jira/browse/SPARK-38042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482963#comment-17482963 ] Johan Nyström-Persson commented on SPARK-38042: --- A similar issue was solve

[jira] [Updated] (SPARK-38042) Encoder cannot be found when a tuple component is a type alias for an Array

[ https://issues.apache.org/jira/browse/SPARK-38042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Nyström-Persson updated SPARK-38042: -- Description: ScalaReflection.dataTypeFor fails when Array[T] has been aliased

[jira] [Created] (SPARK-38042) Encoder cannot be found when a tuple component is a type alias for an Array

Johan Nyström-Persson created SPARK-38042: - Summary: Encoder cannot be found when a tuple component is a type alias for an Array Key: SPARK-38042 URL: https://issues.apache.org/jira/browse/SPARK-38042

[jira] [Commented] (SPARK-37996) Contribution guide is stale

[ https://issues.apache.org/jira/browse/SPARK-37996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482946#comment-17482946 ] Khalid Mammadov commented on SPARK-37996: - Sure, working on it. Thanks for the r