[jira] [Updated] (SPARK-39858) Remove unnecessary AliasHelper or PredicateHelper for some rules

[ https://issues.apache.org/jira/browse/SPARK-39858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-39858: --- Summary: Remove unnecessary AliasHelper or PredicateHelper for some rules (was: Remove unnecessary

[jira] [Created] (SPARK-39858) Remove unnecessary AliasHelper for some rules

jiaan.geng created SPARK-39858: -- Summary: Remove unnecessary AliasHelper for some rules Key: SPARK-39858 URL: https://issues.apache.org/jira/browse/SPARK-39858 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-39837) Filesystem leak when running `TPC-DS queries with SF=1`

[ https://issues.apache.org/jira/browse/SPARK-39837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-39837. -- Resolution: Not A Bug Just close delayed, not leaked.   > Filesystem leak when running `TPC-DS

[jira] [Assigned] (SPARK-39857) V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate

[ https://issues.apache.org/jira/browse/SPARK-39857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39857: Assignee: Apache Spark > V2ExpressionBuilder uses the wrong LiteralValue data type for

[jira] [Commented] (SPARK-39857) V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate

[ https://issues.apache.org/jira/browse/SPARK-39857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570622#comment-17570622 ] Apache Spark commented on SPARK-39857: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-39857) V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate

[ https://issues.apache.org/jira/browse/SPARK-39857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570621#comment-17570621 ] Apache Spark commented on SPARK-39857: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39857) V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate

[ https://issues.apache.org/jira/browse/SPARK-39857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39857: Assignee: (was: Apache Spark) > V2ExpressionBuilder uses the wrong LiteralValue data

[jira] [Created] (SPARK-39857) V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate

Huaxin Gao created SPARK-39857: -- Summary: V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate Key: SPARK-39857 URL: https://issues.apache.org/jira/browse/SPARK-39857 Project:

[jira] [Resolved] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39856. -- Fix Version/s: 3.3.1 3.0.4 3.1.4

[jira] [Assigned] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39856: Assignee: Hyukjin Kwon > Avoid OOM in TPC-DS build with SMJ >

[jira] [Resolved] (SPARK-39840) Factor PythonArrowInput out as a symmetry to PythonArrowOutput

[ https://issues.apache.org/jira/browse/SPARK-39840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39840. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37253

[jira] [Assigned] (SPARK-39840) Factor PythonArrowInput out as a symmetry to PythonArrowOutput

[ https://issues.apache.org/jira/browse/SPARK-39840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39840: Assignee: Hyukjin Kwon > Factor PythonArrowInput out as a symmetry to PythonArrowOutput

[jira] [Commented] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570588#comment-17570588 ] Apache Spark commented on SPARK-39856: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570587#comment-17570587 ] Apache Spark commented on SPARK-39856: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39856: Assignee: Apache Spark > Avoid OOM in TPC-DS build with SMJ >

[jira] [Assigned] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39856: Assignee: (was: Apache Spark) > Avoid OOM in TPC-DS build with SMJ >

[jira] [Created] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

Hyukjin Kwon created SPARK-39856: Summary: Avoid OOM in TPC-DS build with SMJ Key: SPARK-39856 URL: https://issues.apache.org/jira/browse/SPARK-39856 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-39856) Avoid OOM in TPC-DS build with SMJ

[ https://issues.apache.org/jira/browse/SPARK-39856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-39856: - Issue Type: Test (was: Improvement) > Avoid OOM in TPC-DS build with SMJ >

[jira] [Assigned] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

[ https://issues.apache.org/jira/browse/SPARK-39854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39854: Assignee: Apache Spark > Catalyst 'ColumnPruning' Optimizer does not play well with sql

[jira] [Commented] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

[ https://issues.apache.org/jira/browse/SPARK-39854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570540#comment-17570540 ] Apache Spark commented on SPARK-39854: -- User 'jiaji-wu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

[ https://issues.apache.org/jira/browse/SPARK-39854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39854: Assignee: (was: Apache Spark) > Catalyst 'ColumnPruning' Optimizer does not play

[jira] [Commented] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

[ https://issues.apache.org/jira/browse/SPARK-39854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570541#comment-17570541 ] Apache Spark commented on SPARK-39854: -- User 'jiaji-wu' has created a pull request for this issue:

[jira] [Commented] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

[ https://issues.apache.org/jira/browse/SPARK-39854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570535#comment-17570535 ] Jiaji Wu commented on SPARK-39854: -- One workaround is to exclude *ColumnPruning* by set spark config:

[jira] [Commented] (SPARK-39855) Unable to set zstd compression level while writing orc files

[ https://issues.apache.org/jira/browse/SPARK-39855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570504#comment-17570504 ] shezm commented on SPARK-39855: --- I will follow up on this issue > Unable to set zstd compression level

[jira] [Created] (SPARK-39855) Unable to set zstd compression level while writing orc files

shezm created SPARK-39855: - Summary: Unable to set zstd compression level while writing orc files Key: SPARK-39855 URL: https://issues.apache.org/jira/browse/SPARK-39855 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

[ https://issues.apache.org/jira/browse/SPARK-39854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaji Wu updated SPARK-39854: - Affects Version/s: 3.2.1 > Catalyst 'ColumnPruning' Optimizer does not play well with sql function >

[jira] [Created] (SPARK-39854) Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode'

Jiaji Wu created SPARK-39854: Summary: Catalyst 'ColumnPruning' Optimizer does not play well with sql function 'explode' Key: SPARK-39854 URL: https://issues.apache.org/jira/browse/SPARK-39854 Project:

[jira] [Assigned] (SPARK-39853) Support stage level schedule for standalone cluster when dynamic allocation is disabled

[ https://issues.apache.org/jira/browse/SPARK-39853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39853: Assignee: (was: Apache Spark) > Support stage level schedule for standalone cluster

[jira] [Commented] (SPARK-39853) Support stage level schedule for standalone cluster when dynamic allocation is disabled

[ https://issues.apache.org/jira/browse/SPARK-39853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570501#comment-17570501 ] Apache Spark commented on SPARK-39853: -- User 'ivoson' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39853) Support stage level schedule for standalone cluster when dynamic allocation is disabled

[ https://issues.apache.org/jira/browse/SPARK-39853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39853: Assignee: Apache Spark > Support stage level schedule for standalone cluster when

[jira] [Created] (SPARK-39853) Support stage level schedule for standalone cluster when dynamic allocation is disabled

huangtengfei created SPARK-39853: Summary: Support stage level schedule for standalone cluster when dynamic allocation is disabled Key: SPARK-39853 URL: https://issues.apache.org/jira/browse/SPARK-39853

[jira] [Updated] (SPARK-39851) Improve join stats estimation if one side can keep uniqueness

[ https://issues.apache.org/jira/browse/SPARK-39851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-39851: Summary: Improve join stats estimation if one side can keep uniqueness (was: Fix join stats

[jira] [Assigned] (SPARK-39851) Fix join stats estimation if one side can keep uniqueness

[ https://issues.apache.org/jira/browse/SPARK-39851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39851: Assignee: (was: Apache Spark) > Fix join stats estimation if one side can keep

[jira] [Commented] (SPARK-39851) Fix join stats estimation if one side can keep uniqueness

[ https://issues.apache.org/jira/browse/SPARK-39851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570473#comment-17570473 ] Apache Spark commented on SPARK-39851: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39851) Fix join stats estimation if one side can keep uniqueness

[ https://issues.apache.org/jira/browse/SPARK-39851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39851: Assignee: Apache Spark > Fix join stats estimation if one side can keep uniqueness >

[jira] [Commented] (SPARK-39851) Fix join stats estimation if one side can keep uniqueness

[ https://issues.apache.org/jira/browse/SPARK-39851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570474#comment-17570474 ] Apache Spark commented on SPARK-39851: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-39852) Unify v1 and v2 DESCRIBE TABLE tests for columns

[ https://issues.apache.org/jira/browse/SPARK-39852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570468#comment-17570468 ] Apache Spark commented on SPARK-39852: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-39852) Unify v1 and v2 DESCRIBE TABLE tests for columns

[ https://issues.apache.org/jira/browse/SPARK-39852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570467#comment-17570467 ] Apache Spark commented on SPARK-39852: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39852) Unify v1 and v2 DESCRIBE TABLE tests for columns

[ https://issues.apache.org/jira/browse/SPARK-39852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39852: Assignee: Max Gekk (was: Apache Spark) > Unify v1 and v2 DESCRIBE TABLE tests for

[jira] [Assigned] (SPARK-39852) Unify v1 and v2 DESCRIBE TABLE tests for columns

[ https://issues.apache.org/jira/browse/SPARK-39852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39852: Assignee: Apache Spark (was: Max Gekk) > Unify v1 and v2 DESCRIBE TABLE tests for

[jira] [Created] (SPARK-39852) Unify v1 and v2 DESCRIBE TABLE tests for columns

Max Gekk created SPARK-39852: Summary: Unify v1 and v2 DESCRIBE TABLE tests for columns Key: SPARK-39852 URL: https://issues.apache.org/jira/browse/SPARK-39852 Project: Spark Issue Type:

[jira] [Created] (SPARK-39851) Fix join stats estimation if one side can keep uniqueness

Yuming Wang created SPARK-39851: --- Summary: Fix join stats estimation if one side can keep uniqueness Key: SPARK-39851 URL: https://issues.apache.org/jira/browse/SPARK-39851 Project: Spark

[jira] [Assigned] (SPARK-39850) Print applicationId once applied from yarn rm

[ https://issues.apache.org/jira/browse/SPARK-39850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39850: Assignee: (was: Apache Spark) > Print applicationId once applied from yarn rm >

[jira] [Commented] (SPARK-39850) Print applicationId once applied from yarn rm

[ https://issues.apache.org/jira/browse/SPARK-39850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570447#comment-17570447 ] Apache Spark commented on SPARK-39850: -- User 'DongweiLee' has created a pull request for this

[jira] [Assigned] (SPARK-39850) Print applicationId once applied from yarn rm

[ https://issues.apache.org/jira/browse/SPARK-39850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39850: Assignee: Apache Spark > Print applicationId once applied from yarn rm >

[jira] [Created] (SPARK-39850) Print applicationId once applied from yarn rm

LiDongwei created SPARK-39850: - Summary: Print applicationId once applied from yarn rm Key: SPARK-39850 URL: https://issues.apache.org/jira/browse/SPARK-39850 Project: Spark Issue Type:

[jira] [Commented] (SPARK-39849) Dataset.as(StructType) fills missing new columns with null value

[ https://issues.apache.org/jira/browse/SPARK-39849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570443#comment-17570443 ] Apache Spark commented on SPARK-39849: -- User 'c21' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39849) Dataset.as(StructType) fills missing new columns with null value

[ https://issues.apache.org/jira/browse/SPARK-39849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39849: Assignee: (was: Apache Spark) > Dataset.as(StructType) fills missing new columns

[jira] [Commented] (SPARK-39849) Dataset.as(StructType) fills missing new columns with null value

[ https://issues.apache.org/jira/browse/SPARK-39849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570442#comment-17570442 ] Apache Spark commented on SPARK-39849: -- User 'c21' has created a pull request for this issue:

[jira] [Assigned] (SPARK-39849) Dataset.as(StructType) fills missing new columns with null value

[ https://issues.apache.org/jira/browse/SPARK-39849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39849: Assignee: Apache Spark > Dataset.as(StructType) fills missing new columns with null

[jira] [Created] (SPARK-39849) Dataset.as(StructType) fills missing new columns with null value

Cheng Su created SPARK-39849: Summary: Dataset.as(StructType) fills missing new columns with null value Key: SPARK-39849 URL: https://issues.apache.org/jira/browse/SPARK-39849 Project: Spark

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570435#comment-17570435 ] Apache Spark commented on SPARK-39743: -- User 'ming95' has created a pull request for this

[jira] [Assigned] (SPARK-39743) Unable to set zstd compression level while writing parquet files

[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39743: Assignee: (was: Apache Spark) > Unable to set zstd compression level while writing

[jira] [Assigned] (SPARK-39743) Unable to set zstd compression level while writing parquet files

[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39743: Assignee: Apache Spark > Unable to set zstd compression level while writing parquet

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570434#comment-17570434 ] Apache Spark commented on SPARK-39743: -- User 'ming95' has created a pull request for this