[jira] [Resolved] (SPARK-36440) Spark3 fails to read hive table with mixed format

[ https://issues.apache.org/jira/browse/SPARK-36440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Xu resolved SPARK-36440. -- Fix Version/s: 3.2.0 Resolution: Fixed Already fixed by

[jira] [Commented] (SPARK-36440) Spark3 fails to read hive table with mixed format

[ https://issues.apache.org/jira/browse/SPARK-36440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394995#comment-17394995 ] Jason Xu commented on SPARK-36440: -- I found the root cause it's due to 

[jira] [Assigned] (SPARK-36448) Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions

[ https://issues.apache.org/jira/browse/SPARK-36448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36448: Assignee: Apache Spark > Exceptions in NoSuchItemException.scala have to be case classes

[jira] [Commented] (SPARK-36448) Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions

[ https://issues.apache.org/jira/browse/SPARK-36448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394990#comment-17394990 ] Apache Spark commented on SPARK-36448: -- User 'yeshengm' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36448) Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions

[ https://issues.apache.org/jira/browse/SPARK-36448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36448: Assignee: (was: Apache Spark) > Exceptions in NoSuchItemException.scala have to be

[jira] [Commented] (SPARK-36448) Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions

[ https://issues.apache.org/jira/browse/SPARK-36448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394975#comment-17394975 ] Yesheng Ma commented on SPARK-36448: I'd raise a PR shortly. > Exceptions in

[jira] [Created] (SPARK-36448) Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions

Yesheng Ma created SPARK-36448: -- Summary: Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions Key: SPARK-36448 URL: https://issues.apache.org/jira/browse/SPARK-36448

[jira] [Commented] (SPARK-35320) from_json cannot parse maps with timestamp as key

[ https://issues.apache.org/jira/browse/SPARK-35320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394957#comment-17394957 ] Apache Spark commented on SPARK-35320: -- User 'planga82' has created a pull request for this issue:

[jira] [Commented] (SPARK-36447) Avoid inlining non-deterministic With-CTEs

[ https://issues.apache.org/jira/browse/SPARK-36447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394945#comment-17394945 ] Apache Spark commented on SPARK-36447: -- User 'maryannxue' has created a pull request for this

[jira] [Commented] (SPARK-36447) Avoid inlining non-deterministic With-CTEs

[ https://issues.apache.org/jira/browse/SPARK-36447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394944#comment-17394944 ] Apache Spark commented on SPARK-36447: -- User 'maryannxue' has created a pull request for this

[jira] [Assigned] (SPARK-36447) Avoid inlining non-deterministic With-CTEs

[ https://issues.apache.org/jira/browse/SPARK-36447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36447: Assignee: (was: Apache Spark) > Avoid inlining non-deterministic With-CTEs >

[jira] [Assigned] (SPARK-36447) Avoid inlining non-deterministic With-CTEs

[ https://issues.apache.org/jira/browse/SPARK-36447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36447: Assignee: Apache Spark > Avoid inlining non-deterministic With-CTEs >

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394931#comment-17394931 ] Thomas Graves commented on SPARK-36446: --- Is this with the yarn nodemangar recovery enabled?  ie

[jira] [Comment Edited] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy edited comment on SPARK-36446 at 8/6/21, 5:56 PM: --- The

[jira] [Created] (SPARK-36447) Avoid inlining non-deterministic With-CTEs

Wei Xue created SPARK-36447: --- Summary: Avoid inlining non-deterministic With-CTEs Key: SPARK-36447 URL: https://issues.apache.org/jira/browse/SPARK-36447 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394915#comment-17394915 ] Dongjoon Hyun commented on SPARK-36446: --- cc [~tgraves] and [~mridul] > YARN shuffle server

[jira] [Comment Edited] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy edited comment on SPARK-36446 at 8/6/21, 5:06 PM: --- The

[jira] [Comment Edited] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy edited comment on SPARK-36446 at 8/6/21, 5:06 PM: --- The

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394905#comment-17394905 ] Adam Kennedy commented on SPARK-36446: -- The problem was particularly amplified by the Executor

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394873#comment-17394873 ] Adam Kennedy commented on SPARK-36446: -- Note: While I haven't investigated any other shuffle

[jira] [Updated] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-36446: - Summary: YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated

[jira] [Updated] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Kennedy updated SPARK-36446: - Summary: YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated

[jira] [Created] (SPARK-36446) After dynamic deallocation YARN shuffle server restart crashes all jobs

Adam Kennedy created SPARK-36446: Summary: After dynamic deallocation YARN shuffle server restart crashes all jobs Key: SPARK-36446 URL: https://issues.apache.org/jira/browse/SPARK-36446 Project:

[jira] [Resolved] (SPARK-36359) Coalesce drop all expressions after the first non nullable expression

[ https://issues.apache.org/jira/browse/SPARK-36359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-36359. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33590

[jira] [Assigned] (SPARK-36359) Coalesce drop all expressions after the first non nullable expression

[ https://issues.apache.org/jira/browse/SPARK-36359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-36359: --- Assignee: Yuming Wang > Coalesce drop all expressions after the first non nullable

[jira] [Resolved] (SPARK-36423) Randomize blocks within a push request before pushing to improve block merge ratio

[ https://issues.apache.org/jira/browse/SPARK-36423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36423. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36423) Randomize blocks within a push request before pushing to improve block merge ratio

[ https://issues.apache.org/jira/browse/SPARK-36423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36423: --- Assignee: Min Shen > Randomize blocks within a push request before pushing

[jira] [Assigned] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-595: --- Assignee: Yuto Akutsu > Document "local-cluster" mode > - > >

[jira] [Assigned] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-595: -- Assignee: Apache Spark > Document "local-cluster" mode > - > >

[jira] [Resolved] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-595. - Fix Version/s: 3.3.0 3.2.0 Resolution: Fixed > Document "local-cluster"

[jira] [Assigned] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-595: -- Assignee: (was: Apache Spark) > Document "local-cluster" mode >

[jira] [Reopened] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-595: - > Document "local-cluster" mode > - > > Key: SPARK-595 >

[jira] [Commented] (SPARK-36429) JacksonParser should throw exception when data type unsupported.

[ https://issues.apache.org/jira/browse/SPARK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394726#comment-17394726 ] Apache Spark commented on SPARK-36429: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-36429) JacksonParser should throw exception when data type unsupported.

[ https://issues.apache.org/jira/browse/SPARK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394724#comment-17394724 ] Apache Spark commented on SPARK-36429: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-36339) aggsBuffer should collect AggregateExpression in the map range

[ https://issues.apache.org/jira/browse/SPARK-36339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394709#comment-17394709 ] Apache Spark commented on SPARK-36339: -- User 'gaoyajun02' has created a pull request for this

[jira] [Updated] (SPARK-36443) Demote BroadcastJoin causes performance regression and increases OOM risks

[ https://issues.apache.org/jira/browse/SPARK-36443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-36443: - Description:   h2. A test case Use bin/spark-sql with local mode and all other default settings with

[jira] [Updated] (SPARK-36443) Demote BroadcastJoin causes performance regression and increases OOM risks

[ https://issues.apache.org/jira/browse/SPARK-36443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-36443: - Description:   h2. A test case Use bin/spark-sql with local mode and all other default settings with

[jira] [Commented] (SPARK-36429) JacksonParser should throw exception when data type unsupported.

[ https://issues.apache.org/jira/browse/SPARK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394680#comment-17394680 ] Apache Spark commented on SPARK-36429: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-36429) JacksonParser should throw exception when data type unsupported.

[ https://issues.apache.org/jira/browse/SPARK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394679#comment-17394679 ] Apache Spark commented on SPARK-36429: -- User 'ulysses-you' has created a pull request for this

[jira] [Updated] (SPARK-36443) Demote BroadcastJoin causes performance regression and increases OOM risks

[ https://issues.apache.org/jira/browse/SPARK-36443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-36443: - Attachment: image-2021-08-06-17-57-15-765.png > Demote BroadcastJoin causes performance regression and

[jira] [Updated] (SPARK-36443) Demote BroadcastJoin causes performance regression and increases OOM risks

[ https://issues.apache.org/jira/browse/SPARK-36443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-36443: - Attachment: screenshot-1.png > Demote BroadcastJoin causes performance regression and increases OOM

[jira] [Commented] (SPARK-36339) aggsBuffer should collect AggregateExpression in the map range

[ https://issues.apache.org/jira/browse/SPARK-36339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394657#comment-17394657 ] Apache Spark commented on SPARK-36339: -- User 'gaoyajun02' has created a pull request for this

[jira] [Assigned] (SPARK-36445) ANSI type coercion rule for date time operations

[ https://issues.apache.org/jira/browse/SPARK-36445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36445: Assignee: Gengliang Wang (was: Apache Spark) > ANSI type coercion rule for date time

[jira] [Assigned] (SPARK-36445) ANSI type coercion rule for date time operations

[ https://issues.apache.org/jira/browse/SPARK-36445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36445: Assignee: Apache Spark (was: Gengliang Wang) > ANSI type coercion rule for date time

[jira] [Commented] (SPARK-36445) ANSI type coercion rule for date time operations

[ https://issues.apache.org/jira/browse/SPARK-36445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394649#comment-17394649 ] Apache Spark commented on SPARK-36445: -- User 'gengliangwang' has created a pull request for this

[jira] [Created] (SPARK-36445) ANSI type coercion rule for date time operations

Gengliang Wang created SPARK-36445: -- Summary: ANSI type coercion rule for date time operations Key: SPARK-36445 URL: https://issues.apache.org/jira/browse/SPARK-36445 Project: Spark Issue

[jira] [Commented] (SPARK-36444) Remove OptimizeSubqueries from batch of PartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394641#comment-17394641 ] Yuming Wang commented on SPARK-36444: - Another case: {code:scala} sql("create table t1 using parquet

[jira] [Assigned] (SPARK-36339) aggsBuffer should collect AggregateExpression in the map range

[ https://issues.apache.org/jira/browse/SPARK-36339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36339: --- Assignee: gaoyajun02 > aggsBuffer should collect AggregateExpression in the map range >

[jira] [Resolved] (SPARK-36339) aggsBuffer should collect AggregateExpression in the map range

[ https://issues.apache.org/jira/browse/SPARK-36339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36339. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33574

[jira] [Resolved] (SPARK-36098) Group exception messages in core/storage

[ https://issues.apache.org/jira/browse/SPARK-36098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36098. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33530

[jira] [Updated] (SPARK-36427) Scala API: support CrossValidatorModel get standard deviation of metrics for each paramMap

[ https://issues.apache.org/jira/browse/SPARK-36427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-36427: --- Description: This is the parity feature of https://issues.apache.org/jira/browse/SPARK-36425 Note:

[jira] [Issue Comment Deleted] (SPARK-36428) the 'seconds' parameter of 'make_timestamp' should accept integer type

[ https://issues.apache.org/jira/browse/SPARK-36428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-36428: --- Comment: was deleted (was: I will take a look.) > the 'seconds' parameter of 'make_timestamp'

[jira] [Assigned] (SPARK-36428) the 'seconds' parameter of 'make_timestamp' should accept integer type

[ https://issues.apache.org/jira/browse/SPARK-36428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36428: Assignee: Apache Spark > the 'seconds' parameter of 'make_timestamp' should accept

[jira] [Assigned] (SPARK-36428) the 'seconds' parameter of 'make_timestamp' should accept integer type

[ https://issues.apache.org/jira/browse/SPARK-36428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36428: Assignee: Apache Spark > the 'seconds' parameter of 'make_timestamp' should accept

[jira] [Commented] (SPARK-36428) the 'seconds' parameter of 'make_timestamp' should accept integer type

[ https://issues.apache.org/jira/browse/SPARK-36428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394603#comment-17394603 ] Apache Spark commented on SPARK-36428: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36428) the 'seconds' parameter of 'make_timestamp' should accept integer type

[ https://issues.apache.org/jira/browse/SPARK-36428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36428: Assignee: (was: Apache Spark) > the 'seconds' parameter of 'make_timestamp' should

[jira] [Commented] (SPARK-36444) Remove OptimizeSubqueries from batch of PartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394597#comment-17394597 ] Apache Spark commented on SPARK-36444: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36444) Remove OptimizeSubqueries from batch of PartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36444: Assignee: (was: Apache Spark) > Remove OptimizeSubqueries from batch of

[jira] [Assigned] (SPARK-36444) Remove OptimizeSubqueries from batch of PartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36444: Assignee: Apache Spark > Remove OptimizeSubqueries from batch of PartitionPruning >

[jira] [Assigned] (SPARK-35221) Add join hint build side check

[ https://issues.apache.org/jira/browse/SPARK-35221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35221: --- Assignee: XiDuo You > Add join hint build side check > -- > >

[jira] [Resolved] (SPARK-35221) Add join hint build side check

[ https://issues.apache.org/jira/browse/SPARK-35221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35221. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 32355

[jira] [Updated] (SPARK-36444) Remove OptimizeSubqueries from batch of PartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-36444: Description: To support this case: {code:scala} sql( """ |SELECT date_id,

[jira] [Created] (SPARK-36444) Remove OptimizeSubqueries from batch of PartitionPruning

Yuming Wang created SPARK-36444: --- Summary: Remove OptimizeSubqueries from batch of PartitionPruning Key: SPARK-36444 URL: https://issues.apache.org/jira/browse/SPARK-36444 Project: Spark Issue

[jira] [Updated] (SPARK-36360) StreamingSource duplicates appName

[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Description: The StreamingSource includes the appName in its sourceName. This is not

[jira] [Resolved] (SPARK-36429) JacksonParser should throw exception when data type unsupported.

[ https://issues.apache.org/jira/browse/SPARK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36429. - Fix Version/s: 3.2.0 Assignee: jiaan.geng Resolution: Fixed > JacksonParser

[jira] [Commented] (SPARK-36429) JacksonParser should throw exception when data type unsupported.

[ https://issues.apache.org/jira/browse/SPARK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394557#comment-17394557 ] Apache Spark commented on SPARK-36429: -- User 'sarutak' has created a pull request for this issue: