[jira] [Updated] (SPARK-36057) SPIP: Support Customized Kubernetes Schedulers

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36057: Labels: SPIP (was: ) > SPIP: Support Customized Kubernetes Schedulers > -

[jira] [Updated] (SPARK-36057) SPIP: Support Customized Kubernetes Schedulers

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-36057: Summary: SPIP: Support Customized Kubernetes Schedulers (was: Support Customized Kubernetes Sched

[jira] [Resolved] (SPARK-38542) UnsafeHashedRelation should serialize numKeys out

2022-03-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38542. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull re

[jira] [Updated] (SPARK-38324) The second range is not [0, 59] in the day time ANSI interval

2022-03-15 Thread chong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chong updated SPARK-38324: -- Description: [https://spark.apache.org/docs/latest/sql-ref-datatypes.html] * SECOND, seconds within minutes a

[jira] [Updated] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-38558: Fix Version/s: 3.3.0 (was: 3.4.0) > Remove unnecessary casts between Intege

[jira] [Assigned] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38558: --- Assignee: David Cashman > Remove unnecessary casts between IntegerType and IntDecimal > ---

[jira] [Resolved] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38558. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35863 [https://gith

[jira] [Assigned] (SPARK-38564) Support collecting metrics from streaming sinks

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38564: Assignee: Apache Spark > Support collecting metrics from streaming sinks > --

[jira] [Assigned] (SPARK-38564) Support collecting metrics from streaming sinks

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38564: Assignee: (was: Apache Spark) > Support collecting metrics from streaming sinks > ---

[jira] [Commented] (SPARK-38564) Support collecting metrics from streaming sinks

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507359#comment-17507359 ] Apache Spark commented on SPARK-38564: -- User 'jerrypeng' has created a pull request

[jira] [Created] (SPARK-38564) Support collecting metrics from streaming sinks

2022-03-15 Thread Boyang Jerry Peng (Jira)
Boyang Jerry Peng created SPARK-38564: - Summary: Support collecting metrics from streaming sinks Key: SPARK-38564 URL: https://issues.apache.org/jira/browse/SPARK-38564 Project: Spark Iss

[jira] [Commented] (SPARK-38563) Upgrade to Py4J 0.10.9.4

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507354#comment-17507354 ] Apache Spark commented on SPARK-38563: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-38563) Upgrade to Py4J 0.10.9.4

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38563: Assignee: (was: Apache Spark) > Upgrade to Py4J 0.10.9.4 > >

[jira] [Assigned] (SPARK-38563) Upgrade to Py4J 0.10.9.4

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38563: Assignee: Apache Spark > Upgrade to Py4J 0.10.9.4 > > >

[jira] [Commented] (SPARK-38563) Upgrade to Py4J 0.10.9.4

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507352#comment-17507352 ] Apache Spark commented on SPARK-38563: -- User 'HyukjinKwon' has created a pull reque

[jira] [Created] (SPARK-38563) Upgrade to Py4J 0.10.9.4

2022-03-15 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-38563: Summary: Upgrade to Py4J 0.10.9.4 Key: SPARK-38563 URL: https://issues.apache.org/jira/browse/SPARK-38563 Project: Spark Issue Type: Bug Components

[jira] [Assigned] (SPARK-38562) Add doc for Volcano scheduler

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38562: Assignee: Apache Spark > Add doc for Volcano scheduler > - >

[jira] [Commented] (SPARK-38562) Add doc for Volcano scheduler

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507334#comment-17507334 ] Apache Spark commented on SPARK-38562: -- User 'Yikun' has created a pull request for

[jira] [Commented] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507335#comment-17507335 ] Apache Spark commented on SPARK-38561: -- User 'Yikun' has created a pull request for

[jira] [Commented] (SPARK-38562) Add doc for Volcano scheduler

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507336#comment-17507336 ] Apache Spark commented on SPARK-38562: -- User 'Yikun' has created a pull request for

[jira] [Assigned] (SPARK-38562) Add doc for Volcano scheduler

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38562: Assignee: (was: Apache Spark) > Add doc for Volcano scheduler > -

[jira] [Assigned] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38561: Assignee: (was: Apache Spark) > Add doc for "Customized Kubernetes Schedulers" >

[jira] [Commented] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507333#comment-17507333 ] Apache Spark commented on SPARK-38561: -- User 'Yikun' has created a pull request for

[jira] [Assigned] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38561: Assignee: Apache Spark > Add doc for "Customized Kubernetes Schedulers" > ---

[jira] [Created] (SPARK-38562) Add doc for Volcano scheduler

2022-03-15 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38562: --- Summary: Add doc for Volcano scheduler Key: SPARK-38562 URL: https://issues.apache.org/jira/browse/SPARK-38562 Project: Spark Issue Type: Sub-task Co

[jira] [Created] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-15 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38561: --- Summary: Add doc for "Customized Kubernetes Schedulers" Key: SPARK-38561 URL: https://issues.apache.org/jira/browse/SPARK-38561 Project: Spark Issue Type: Sub-

[jira] [Resolved] (SPARK-38424) Disallow unused casts and ignores

2022-03-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38424. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35740 [https://gi

[jira] [Assigned] (SPARK-38424) Disallow unused casts and ignores

2022-03-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38424: Assignee: Maciej Szymkiewicz > Disallow unused casts and ignores > --

[jira] [Created] (SPARK-38560) If `Sum`, `Count` with distinct, cannot do partial agg push down.

2022-03-15 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-38560: -- Summary: If `Sum`, `Count` with distinct, cannot do partial agg push down. Key: SPARK-38560 URL: https://issues.apache.org/jira/browse/SPARK-38560 Project: Spark

[jira] [Commented] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507323#comment-17507323 ] Apache Spark commented on SPARK-38559: -- User 'caican00' has created a pull request

[jira] [Commented] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507325#comment-17507325 ] Apache Spark commented on SPARK-38559: -- User 'caican00' has created a pull request

[jira] [Assigned] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38559: Assignee: (was: Apache Spark) > display the number of empty partitions on spark ui >

[jira] [Assigned] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38559: Assignee: Apache Spark > display the number of empty partitions on spark ui > ---

[jira] [Updated] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Description: When demoting join from broadcast-hash to smj, i think it is necessary to display the number o

[jira] [Updated] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Description: When demoting join from broadcast-hash to smj, i think it is necessary to display the number o

[jira] [Updated] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Attachment: image-2022-03-16-11-07-39-182.png > display the number of empty partitions on spark ui > ---

[jira] [Updated] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Description: When demoting join from broadcast-hash to smj, i think it is necessary to display the number o

[jira] [Updated] (SPARK-38559) display the number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Summary: display the number of empty partitions on spark ui (was: display number of empty partitions on spa

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Attachment: (was: ui.png) > display number of empty partitions on spark ui > ---

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Description: When demoting join from broadcast-hash to smj, i think it is necessary to display number of em

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Attachment: image-2022-03-16-10-56-46-446.png > display number of empty partitions on spark ui > ---

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Attachment: ui.png > display number of empty partitions on spark ui > --

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Attachment: (was: 小米办公20220316-105510.png) > display number of empty partitions on spark ui > --

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Attachment: 小米办公20220316-105510.png > display number of empty partitions on spark ui > -

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Summary: display number of empty partitions on spark ui (was: display number of empty partitions on spark u

[jira] [Updated] (SPARK-38527) Set the minimum Volcano version

2022-03-15 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-38527: Labels: release-notes (was: ) > Set the minimum Volcano version > --- > >

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui when demoting join from broadcast-hash to smj

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Description: When demoting join from broadcast-hash to smj, i think it is necessary to display number of em

[jira] [Created] (SPARK-38559) show number of empty partitions on spark ui when demoting join from broadcast-hash to smj

2022-03-15 Thread caican (Jira)
caican created SPARK-38559: -- Summary: show number of empty partitions on spark ui when demoting join from broadcast-hash to smj Key: SPARK-38559 URL: https://issues.apache.org/jira/browse/SPARK-38559 Project

[jira] [Updated] (SPARK-38559) display number of empty partitions on spark ui when demoting join from broadcast-hash to smj

2022-03-15 Thread caican (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-38559: --- Summary: display number of empty partitions on spark ui when demoting join from broadcast-hash to smj (was:

[jira] [Resolved] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38508. --- Fix Version/s: 3.3.0 Assignee: Yikun Jiang Resolution: Fixed > Volcano featu

[jira] [Commented] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507314#comment-17507314 ] Dongjoon Hyun commented on SPARK-38508: --- Thanks! > Volcano feature doesn't work o

[jira] [Resolved] (SPARK-38515) Volcano queue is not deleted

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38515. --- Fix Version/s: 3.3.0 Assignee: Yikun Jiang Resolution: Fixed > Volcano queue

[jira] [Commented] (SPARK-38515) Volcano queue is not deleted

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507313#comment-17507313 ] Dongjoon Hyun commented on SPARK-38515: --- Thanks! > Volcano queue is not deleted >

[jira] [Updated] (SPARK-38397) Support Kueue: K8s-native Job Queueing

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38397: -- Affects Version/s: 3.4.0 (was: 3.3.0) > Support Kueue: K8s-native J

[jira] [Updated] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread David Cashman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Cashman updated SPARK-38558: -- Priority: Minor (was: Major) > Remove unnecessary casts between IntegerType and IntDecimal >

[jira] [Assigned] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38558: Assignee: (was: Apache Spark) > Remove unnecessary casts between IntegerType and IntD

[jira] [Assigned] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38558: Assignee: Apache Spark > Remove unnecessary casts between IntegerType and IntDecimal > --

[jira] [Commented] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507300#comment-17507300 ] Apache Spark commented on SPARK-38558: -- User 'cashmand' has created a pull request

[jira] [Created] (SPARK-38558) Remove unnecessary casts between IntegerType and IntDecimal

2022-03-15 Thread David Cashman (Jira)
David Cashman created SPARK-38558: - Summary: Remove unnecessary casts between IntegerType and IntDecimal Key: SPARK-38558 URL: https://issues.apache.org/jira/browse/SPARK-38558 Project: Spark

[jira] [Updated] (SPARK-38529) Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators

2022-03-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-38529: Summary: Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators (was: Gene

[jira] [Updated] (SPARK-38529) Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators

2022-03-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-38529: Issue Type: Improvement (was: Bug) > Prevent GeneratorNestedColumnAliasing to be applied to non-E

[jira] [Commented] (SPARK-38530) GeneratorNestedColumnAliasing does not work correctly for some expressions

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507294#comment-17507294 ] Apache Spark commented on SPARK-38530: -- User 'minyyy' has created a pull request fo

[jira] [Assigned] (SPARK-38530) GeneratorNestedColumnAliasing does not work correctly for some expressions

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38530: Assignee: (was: Apache Spark) > GeneratorNestedColumnAliasing does not work correctly

[jira] [Assigned] (SPARK-38530) GeneratorNestedColumnAliasing does not work correctly for some expressions

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38530: Assignee: Apache Spark > GeneratorNestedColumnAliasing does not work correctly for some e

[jira] [Commented] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2022-03-15 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507293#comment-17507293 ] leesf commented on SPARK-29091: --- any updates here? we also encountered this problem. > sp

[jira] [Assigned] (SPARK-38106) Use error classes in the parsing errors of functions

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38106: Assignee: Apache Spark > Use error classes in the parsing errors of functions > -

[jira] [Assigned] (SPARK-38106) Use error classes in the parsing errors of functions

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38106: Assignee: (was: Apache Spark) > Use error classes in the parsing errors of functions

[jira] [Commented] (SPARK-38106) Use error classes in the parsing errors of functions

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507280#comment-17507280 ] Apache Spark commented on SPARK-38106: -- User 'ivoson' has created a pull request fo

[jira] [Commented] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507278#comment-17507278 ] Yikun Jiang commented on SPARK-38508: - Resolved by https://github.com/apache/spark/p

[jira] [Commented] (SPARK-38515) Volcano queue is not deleted

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507279#comment-17507279 ] Yikun Jiang commented on SPARK-38515: - Resolved by https://github.com/apache/spark/p

[jira] [Comment Edited] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507276#comment-17507276 ] Yikun Jiang edited comment on SPARK-38508 at 3/16/22, 12:42 AM: --

[jira] [Comment Edited] (SPARK-38515) Volcano queue is not deleted

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507277#comment-17507277 ] Yikun Jiang edited comment on SPARK-38515 at 3/16/22, 12:40 AM: --

[jira] [Commented] (SPARK-38515) Volcano queue is not deleted

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507277#comment-17507277 ] Yikun Jiang commented on SPARK-38515: - [~dongjoon] Yes > Volcano queue is not delet

[jira] [Commented] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-15 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507276#comment-17507276 ] Yikun Jiang commented on SPARK-38508: - [~dongjoon] Yes > Volcano feature doesn't wo

[jira] [Commented] (SPARK-38531) "Prune unrequired child index" branch of ColumnPruning has wrong condition

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507268#comment-17507268 ] Apache Spark commented on SPARK-38531: -- User 'minyyy' has created a pull request fo

[jira] [Assigned] (SPARK-38531) "Prune unrequired child index" branch of ColumnPruning has wrong condition

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38531: Assignee: (was: Apache Spark) > "Prune unrequired child index" branch of ColumnPrunin

[jira] [Commented] (SPARK-38531) "Prune unrequired child index" branch of ColumnPruning has wrong condition

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507266#comment-17507266 ] Apache Spark commented on SPARK-38531: -- User 'minyyy' has created a pull request fo

[jira] [Assigned] (SPARK-38531) "Prune unrequired child index" branch of ColumnPruning has wrong condition

2022-03-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38531: Assignee: Apache Spark > "Prune unrequired child index" branch of ColumnPruning has wrong

[jira] [Updated] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38329: - Affects Version/s: 3.2.1 (was: 2.4.6) > High I/O wait when Spark Stru

[jira] [Commented] (SPARK-38557) What may be a cause for HDFSMetadataCommitter: Error while fetching MetaData and how to fix or work around this?

2022-03-15 Thread Dmitry Goldenberg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507263#comment-17507263 ] Dmitry Goldenberg commented on SPARK-38557: --- Likely a DUP of https://github.co

[jira] [Created] (SPARK-38557) What may be a cause for HDFSMetadataCommitter: Error while fetching MetaData and how to fix or work around this?

2022-03-15 Thread Dmitry Goldenberg (Jira)
Dmitry Goldenberg created SPARK-38557: - Summary: What may be a cause for HDFSMetadataCommitter: Error while fetching MetaData and how to fix or work around this? Key: SPARK-38557 URL: https://issues.apache.org

[jira] [Commented] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

2022-03-15 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507257#comment-17507257 ] Mridul Muralidharan commented on SPARK-38388: - Agree with [~jiangxb1987], ei

[jira] [Commented] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507241#comment-17507241 ] Neven Jovic commented on SPARK-38329: - [~hyukjin.kwon]  I updated Spark to 3.2.1, an

[jira] [Updated] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neven Jovic updated SPARK-38329: Attachment: Screenshot from 2022-02-25 14-16-11.png > High I/O wait when Spark Structured Streamin

[jira] [Comment Edited] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507241#comment-17507241 ] Neven Jovic edited comment on SPARK-38329 at 3/15/22, 9:41 PM: ---

[jira] [Updated] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neven Jovic updated SPARK-38329: Attachment: (was: 100k_zbx_21.png) > High I/O wait when Spark Structured Streaming checkpoint

[jira] [Updated] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neven Jovic updated SPARK-38329: Attachment: q.png > High I/O wait when Spark Structured Streaming checkpoint changed to EFS >

[jira] [Comment Edited] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507241#comment-17507241 ] Neven Jovic edited comment on SPARK-38329 at 3/15/22, 9:40 PM: ---

[jira] [Comment Edited] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507241#comment-17507241 ] Neven Jovic edited comment on SPARK-38329 at 3/15/22, 9:40 PM: ---

[jira] [Updated] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neven Jovic updated SPARK-38329: Attachment: (was: Screenshot from 2022-02-25 14-16-11.png) > High I/O wait when Spark Structur

[jira] [Updated] (SPARK-38329) High I/O wait when Spark Structured Streaming checkpoint changed to EFS

2022-03-15 Thread Neven Jovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neven Jovic updated SPARK-38329: Attachment: 100k_zbx_21.png > High I/O wait when Spark Structured Streaming checkpoint changed to

[jira] [Resolved] (SPARK-38204) All state operators are at a risk of inconsistency between state partitioning and operator partitioning

2022-03-15 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li resolved SPARK-38204. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35673 [https://gith

[jira] [Assigned] (SPARK-38204) All state operators are at a risk of inconsistency between state partitioning and operator partitioning

2022-03-15 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li reassigned SPARK-38204: --- Assignee: Jungtaek Lim > All state operators are at a risk of inconsistency between state p

[jira] [Comment Edited] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-15 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503782#comment-17503782 ] Brian Schaefer edited comment on SPARK-38483 at 3/15/22, 8:11 PM:

[jira] [Comment Edited] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-15 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503782#comment-17503782 ] Brian Schaefer edited comment on SPARK-38483 at 3/15/22, 8:11 PM:

[jira] [Updated] (SPARK-38334) Implement support for DEFAULT values for columns in tables

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38334: -- Affects Version/s: 3.4.0 (was: 3.3.0) > Implement support for DEFAU

[jira] [Updated] (SPARK-38335) Parser changes for DEFAULT column support

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38335: -- Fix Version/s: 3.4.0 (was: 3.3.0) > Parser changes for DEFAULT column s

[jira] [Updated] (SPARK-38453) Add volcano section to K8s IT README.md

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38453: -- Component/s: Documentation > Add volcano section to K8s IT README.md > ---

[jira] [Commented] (SPARK-38553) Bump minimum Volcano version to v1.5.1

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507147#comment-17507147 ] Dongjoon Hyun commented on SPARK-38553: --- Since this is a documentation JIRA, I add

[jira] [Updated] (SPARK-38553) Bump minimum Volcano version to v1.5.1

2022-03-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38553: -- Component/s: Documentation > Bump minimum Volcano version to v1.5.1 >

  1   2   >