[jira] [Created] (SPARK-48004) Add WriteFilesExecBase trait for v1 write

2024-04-26 Thread XiDuo You (Jira)
XiDuo You created SPARK-48004: - Summary: Add WriteFilesExecBase trait for v1 write Key: SPARK-48004 URL: https://issues.apache.org/jira/browse/SPARK-48004 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-47285) AdaptiveSparkPlanExec should always use the context.session

2024-03-05 Thread XiDuo You (Jira)
XiDuo You created SPARK-47285: - Summary: AdaptiveSparkPlanExec should always use the context.session Key: SPARK-47285 URL: https://issues.apache.org/jira/browse/SPARK-47285 Project: Spark Issue

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-03-05 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-47177: -- Fix Version/s: 3.4.3 > Cached SQL plan do not display final AQE plan in explain string >

[jira] [Resolved] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-03-04 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-47177. --- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-03-04 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-47177: - Assignee: XiDuo You > Cached SQL plan do not display final AQE plan in explain string >

[jira] [Resolved] (SPARK-47187) Fix hive compress output config does not work

2024-02-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-47187. --- Fix Version/s: 3.4.3 Resolution: Fixed Issue resolved by pull request 45286

[jira] [Assigned] (SPARK-47187) Fix hive compress output config does not work

2024-02-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-47187: - Assignee: XiDuo You > Fix hive compress output config does not work >

[jira] [Created] (SPARK-47187) Fix hive compress output config does not work

2024-02-27 Thread XiDuo You (Jira)
XiDuo You created SPARK-47187: - Summary: Fix hive compress output config does not work Key: SPARK-47187 URL: https://issues.apache.org/jira/browse/SPARK-47187 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-46756) Add rule to rewrite null safe equality join keys

2024-01-17 Thread XiDuo You (Jira)
XiDuo You created SPARK-46756: - Summary: Add rule to rewrite null safe equality join keys Key: SPARK-46756 URL: https://issues.apache.org/jira/browse/SPARK-46756 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46480) Fix NPE when table cache task attempt

2023-12-22 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46480: -- Fix Version/s: 3.5.1 > Fix NPE when table cache task attempt > -

[jira] [Updated] (SPARK-46480) Fix NPE when table cache task attempt

2023-12-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46480: -- Summary: Fix NPE when table cache task attempt (was: Fix NPE when table cache task do attempt) >

[jira] [Updated] (SPARK-46480) Fix NPE when table cache task do attempt

2023-12-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46480: -- Component/s: Spark Core > Fix NPE when table cache task do attempt >

[jira] [Updated] (SPARK-46480) Fix NPE when table cache task do attempt

2023-12-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46480: -- Issue Type: Bug (was: Improvement) > Fix NPE when table cache task do attempt >

[jira] [Created] (SPARK-46480) Fix NPE when table cache task do attempt

2023-12-21 Thread XiDuo You (Jira)
XiDuo You created SPARK-46480: - Summary: Fix NPE when table cache task do attempt Key: SPARK-46480 URL: https://issues.apache.org/jira/browse/SPARK-46480 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-46227) Move `withSQLConf` from SQLHelper trait to `SQLConfHelper` trait

2023-12-03 Thread XiDuo You (Jira)
XiDuo You created SPARK-46227: - Summary: Move `withSQLConf` from SQLHelper trait to `SQLConfHelper` trait Key: SPARK-46227 URL: https://issues.apache.org/jira/browse/SPARK-46227 Project: Spark

[jira] [Assigned] (SPARK-46170) Support inject adaptive query post planner strategy rules in SparkSessionExtensions

2023-11-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-46170: - Assignee: XiDuo You > Support inject adaptive query post planner strategy rules in >

[jira] [Resolved] (SPARK-46170) Support inject adaptive query post planner strategy rules in SparkSessionExtensions

2023-11-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-46170. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44074

[jira] [Created] (SPARK-46170) Support inject adaptive query post planner strategy rules in SparkSessionExtensions

2023-11-28 Thread XiDuo You (Jira)
XiDuo You created SPARK-46170: - Summary: Support inject adaptive query post planner strategy rules in SparkSessionExtensions Key: SPARK-46170 URL: https://issues.apache.org/jira/browse/SPARK-46170

[jira] [Commented] (SPARK-46105) df.emptyDataFrame shows 1 if we repartition(1) in Spark 3.3.x and above

2023-11-26 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789908#comment-17789908 ] XiDuo You commented on SPARK-46105: --- Please see SPARK-39915 > df.emptyDataFrame shows 1 if we

[jira] [Updated] (SPARK-46090) Support plan fragment level SQL configs in AQE

2023-11-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46090: -- Summary: Support plan fragment level SQL configs in AQE (was: Support plan fragment level SQL

[jira] [Updated] (SPARK-46090) Support plan fragment level SQL configs

2023-11-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46090: -- Summary: Support plan fragment level SQL configs (was: Support stage level SQL configs) > Support

[jira] [Updated] (SPARK-46090) Support plan fragment level SQL configs

2023-11-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-46090: -- Description: AQE executes query plan stage by stage, so there is a chance to support plan fragment

[jira] [Created] (SPARK-46090) Support stage level SQL configs

2023-11-24 Thread XiDuo You (Jira)
XiDuo You created SPARK-46090: - Summary: Support stage level SQL configs Key: SPARK-46090 URL: https://issues.apache.org/jira/browse/SPARK-46090 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45882) BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning

2023-11-14 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-45882: -- Fix Version/s: 3.4.2 4.0.0 > BroadcastHashJoinExec propagate partitioning should

[jira] [Resolved] (SPARK-45882) BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning

2023-11-14 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-45882. --- Fix Version/s: 3.5.1 Resolution: Fixed Issue resolved by pull request 43792

[jira] [Assigned] (SPARK-45882) BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning

2023-11-14 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-45882: - Assignee: XiDuo You > BroadcastHashJoinExec propagate partitioning should respect >

[jira] [Created] (SPARK-45882) BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning

2023-11-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-45882: - Summary: BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning Key: SPARK-45882 URL: https://issues.apache.org/jira/browse/SPARK-45882

[jira] [Resolved] (SPARK-34444) Pushdown scalar-subquery filter to FileSourceScan

2023-11-06 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-3. --- Fix Version/s: 4.0.0 Resolution: Fixed > Pushdown scalar-subquery filter to FileSourceScan >

[jira] [Updated] (SPARK-45740) Relax the node prefix of SparkPlanGraphCluster

2023-10-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-45740: -- Summary: Relax the node prefix of SparkPlanGraphCluster (was: Release the node prefix of

[jira] [Created] (SPARK-45740) Release the node prefix of SparkPlanGraphCluster

2023-10-31 Thread XiDuo You (Jira)
XiDuo You created SPARK-45740: - Summary: Release the node prefix of SparkPlanGraphCluster Key: SPARK-45740 URL: https://issues.apache.org/jira/browse/SPARK-45740 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-45705) Fix flaky test: Status of a failed DDL/DML with no jobs should be FAILED

2023-10-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-45705. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43554

[jira] [Assigned] (SPARK-45632) Table cache should avoid unnecessary ColumnarToRow when enable AQE

2023-10-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-45632: - Assignee: XiDuo You > Table cache should avoid unnecessary ColumnarToRow when enable AQE >

[jira] [Resolved] (SPARK-45632) Table cache should avoid unnecessary ColumnarToRow when enable AQE

2023-10-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-45632. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43484

[jira] [Created] (SPARK-45632) Table cache should avoid unnecessary ColumnarToRow when enable AQE

2023-10-23 Thread XiDuo You (Jira)
XiDuo You created SPARK-45632: - Summary: Table cache should avoid unnecessary ColumnarToRow when enable AQE Key: SPARK-45632 URL: https://issues.apache.org/jira/browse/SPARK-45632 Project: Spark

[jira] [Commented] (SPARK-45443) Revisit TableCacheQueryStage to avoid replicated InMemoryRelation materialization

2023-10-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773507#comment-17773507 ] XiDuo You commented on SPARK-45443: --- > But this idea only work for one query Please see the following

[jira] [Commented] (SPARK-45443) Revisit TableCacheQueryStage to avoid replicated InMemoryRelation materialization

2023-10-08 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773074#comment-17773074 ] XiDuo You commented on SPARK-45443: --- > Can this increase probability of concurrent IMR materialization

[jira] [Created] (SPARK-45451) Make the default storage level of dataset cache configurable

2023-10-07 Thread XiDuo You (Jira)
XiDuo You created SPARK-45451: - Summary: Make the default storage level of dataset cache configurable Key: SPARK-45451 URL: https://issues.apache.org/jira/browse/SPARK-45451 Project: Spark

[jira] [Commented] (SPARK-45443) Revisit TableCacheQueryStage to avoid replicated InMemoryRelation materialization

2023-10-06 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772729#comment-17772729 ] XiDuo You commented on SPARK-45443: --- hi [~erenavsarogullari] , it seems that, it depends on the

[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-09-26 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769383#comment-17769383 ] XiDuo You commented on SPARK-45282: --- I can not re-produce this issue in master branch (4.0.0),

[jira] [Assigned] (SPARK-45244) Correct spelling in VolcanoTestsSuite

2023-09-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-45244: - Assignee: Binjie Yang > Correct spelling in VolcanoTestsSuite >

[jira] [Resolved] (SPARK-45244) Correct spelling in VolcanoTestsSuite

2023-09-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-45244. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43026

[jira] [Resolved] (SPARK-45191) InMemoryTableScanExec simpleStringWithNodeId adds columnar info

2023-09-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-45191. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42967

[jira] [Created] (SPARK-45191) InMemoryTableScanExec simpleStringWithNodeId adds columnar info

2023-09-17 Thread XiDuo You (Jira)
XiDuo You created SPARK-45191: - Summary: InMemoryTableScanExec simpleStringWithNodeId adds columnar info Key: SPARK-45191 URL: https://issues.apache.org/jira/browse/SPARK-45191 Project: Spark

[jira] [Commented] (SPARK-44598) spark 3.2+ can not read hive table with hbase serde when hbase StorefileSize is 0

2023-07-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749421#comment-17749421 ] XiDuo You commented on SPARK-44598: --- please try `--conf spark.hadoopRDD.ignoreEmptySplits=false` >

[jira] [Assigned] (SPARK-44579) Support Interrupt On Cancel in SQLExecution

2023-07-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-44579: - Assignee: Kent Yao > Support Interrupt On Cancel in SQLExecution >

[jira] [Resolved] (SPARK-44579) Support Interrupt On Cancel in SQLExecution

2023-07-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-44579. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42199

[jira] [Resolved] (SPARK-43402) FileSourceScanExec supports push down data filter with scalar subquery

2023-07-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-43402. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41088

[jira] [Assigned] (SPARK-43402) FileSourceScanExec supports push down data filter with scalar subquery

2023-07-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-43402: - Assignee: XiDuo You > FileSourceScanExec supports push down data filter with scalar subquery >

[jira] [Commented] (SPARK-43777) Coalescing partitions in AQE returns different results with row_number windows.

2023-05-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726025#comment-17726025 ] XiDuo You commented on SPARK-43777: --- It acutally is a random event that all the row count of id 2 are

[jira] [Created] (SPARK-43420) Make DisableUnnecessaryBucketedScan smart with table cache

2023-05-09 Thread XiDuo You (Jira)
XiDuo You created SPARK-43420: - Summary: Make DisableUnnecessaryBucketedScan smart with table cache Key: SPARK-43420 URL: https://issues.apache.org/jira/browse/SPARK-43420 Project: Spark Issue

[jira] [Created] (SPARK-43402) FileSourceScanExec supports push down data filter with scalar subquery

2023-05-07 Thread XiDuo You (Jira)
XiDuo You created SPARK-43402: - Summary: FileSourceScanExec supports push down data filter with scalar subquery Key: SPARK-43402 URL: https://issues.apache.org/jira/browse/SPARK-43402 Project: Spark

[jira] [Updated] (SPARK-43377) Enable spark.sql.thriftServer.interruptOnCancel by default

2023-05-04 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43377: -- Summary: Enable spark.sql.thriftServer.interruptOnCancel by default (was: Enable

[jira] [Created] (SPARK-43377) Enable spark.sql.thriftServer.interruptOnCancel by defauly

2023-05-04 Thread XiDuo You (Jira)
XiDuo You created SPARK-43377: - Summary: Enable spark.sql.thriftServer.interruptOnCancel by defauly Key: SPARK-43377 URL: https://issues.apache.org/jira/browse/SPARK-43377 Project: Spark Issue

[jira] [Created] (SPARK-43376) Improve reuse subquery with table cache

2023-05-04 Thread XiDuo You (Jira)
XiDuo You created SPARK-43376: - Summary: Improve reuse subquery with table cache Key: SPARK-43376 URL: https://issues.apache.org/jira/browse/SPARK-43376 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-43317) Support combine adjacent aggregation

2023-04-28 Thread XiDuo You (Jira)
XiDuo You created SPARK-43317: - Summary: Support combine adjacent aggregation Key: SPARK-43317 URL: https://issues.apache.org/jira/browse/SPARK-43317 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-43281) Fix concurrent writer does not update file metrics

2023-04-25 Thread XiDuo You (Jira)
XiDuo You created SPARK-43281: - Summary: Fix concurrent writer does not update file metrics Key: SPARK-43281 URL: https://issues.apache.org/jira/browse/SPARK-43281 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance for high cardinality

2023-04-23 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has three preformance issues: - heavy overhead of scala

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance for high cardinality

2023-04-23 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Summary: Improve ObjectHashAggregateExec performance for high cardinality (was: Improve

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance with high cardinality

2023-04-23 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Summary: Improve ObjectHashAggregateExec performance with high cardinality (was: Improve

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance

2023-04-23 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has three preformance issues: - heavy overhead of scala

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance

2023-04-23 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has three preformance issues: - heavy overhead of scala

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance

2023-04-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has two preformance issues: - heavy overhead of scala

[jira] [Created] (SPARK-43232) Improve ObjectHashAggregateExec performance

2023-04-21 Thread XiDuo You (Jira)
XiDuo You created SPARK-43232: - Summary: Improve ObjectHashAggregateExec performance Key: SPARK-43232 URL: https://issues.apache.org/jira/browse/SPARK-43232 Project: Spark Issue Type:

[jira] [Created] (SPARK-43026) Apply AQE with non-exchange table cache

2023-04-04 Thread XiDuo You (Jira)
XiDuo You created SPARK-43026: - Summary: Apply AQE with non-exchange table cache Key: SPARK-43026 URL: https://issues.apache.org/jira/browse/SPARK-43026 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-42963) Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

2023-03-29 Thread XiDuo You (Jira)
XiDuo You created SPARK-42963: - Summary: Extend SparkSessionExtensions to inject rules into AQE query stage optimizer Key: SPARK-42963 URL: https://issues.apache.org/jira/browse/SPARK-42963 Project:

[jira] [Created] (SPARK-42942) Support coalesce table cache stage partitions

2023-03-27 Thread XiDuo You (Jira)
XiDuo You created SPARK-42942: - Summary: Support coalesce table cache stage partitions Key: SPARK-42942 URL: https://issues.apache.org/jira/browse/SPARK-42942 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42815) Subexpression elimination support shortcut expression

2023-03-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42815: -- Summary: Subexpression elimination support shortcut expression (was: Subexpression elimination

[jira] [Updated] (SPARK-42815) Subexpression elimination support shortcut conditional expression

2023-03-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42815: -- Description: The subexpression may not need to eval even if it appears more than once. e.g., 

[jira] [Created] (SPARK-42815) Subexpression elimination support shortcut conditional expression

2023-03-15 Thread XiDuo You (Jira)
XiDuo You created SPARK-42815: - Summary: Subexpression elimination support shortcut conditional expression Key: SPARK-42815 URL: https://issues.apache.org/jira/browse/SPARK-42815 Project: Spark

[jira] [Created] (SPARK-42778) QueryStageExec should respect supportsRowBased

2023-03-14 Thread XiDuo You (Jira)
XiDuo You created SPARK-42778: - Summary: QueryStageExec should respect supportsRowBased Key: SPARK-42778 URL: https://issues.apache.org/jira/browse/SPARK-42778 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42768) Enable cached plan apply AQE by default

2023-03-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42768: -- Summary: Enable cached plan apply AQE by default (was: Enable cache apply AQE by default) > Enable

[jira] [Created] (SPARK-42768) Enable cache apply AQE by default

2023-03-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-42768: - Summary: Enable cache apply AQE by default Key: SPARK-42768 URL: https://issues.apache.org/jira/browse/SPARK-42768 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696070#comment-17696070 ] XiDuo You commented on SPARK-42650: --- To be clear, it is the issue of Spark 3.2.3. Spark3.2.1, 3.3.x

[jira] [Updated] (SPARK-42651) Optimize global sort to driver sort

2023-03-02 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42651: -- Description: If the size of plan is small enough, it's more efficient to sort all rows at driver side

[jira] [Created] (SPARK-42651) Optimize global sort to driver sort

2023-03-02 Thread XiDuo You (Jira)
XiDuo You created SPARK-42651: - Summary: Optimize global sort to driver sort Key: SPARK-42651 URL: https://issues.apache.org/jira/browse/SPARK-42651 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-39316) Merge PromotePrecision and CheckOverflow into decimal binary arithmetic

2023-03-01 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-39316: -- Description: Fix the bug of `TypeCoercion`, for example: {code:java} SELECT CAST(1 AS DECIMAL(28,

[jira] [Updated] (SPARK-42548) Add ReferenceAllColumns to skip rewriting attributes

2023-02-23 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42548: -- Summary: Add ReferenceAllColumns to skip rewriting attributes (was: Add PlainReferences to skip

[jira] [Created] (SPARK-42548) Add PlainReferences to skip rewriting attributes

2023-02-23 Thread XiDuo You (Jira)
XiDuo You created SPARK-42548: - Summary: Add PlainReferences to skip rewriting attributes Key: SPARK-42548 URL: https://issues.apache.org/jira/browse/SPARK-42548 Project: Spark Issue Type:

[jira] [Commented] (SPARK-40278) Used databricks spark-sql-pref with Spark 3.3 to run 3TB tpcds test failed

2023-02-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691861#comment-17691861 ] XiDuo You commented on SPARK-40278: --- It should work now(3.4). We figure out SQL execution status by

[jira] [Updated] (SPARK-42504) NestedColumnAliasing support pruning adjacent projects

2023-02-20 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42504: -- Description: CollapseProject won't combine adjacent projects into one, e.g. non-cheap expression has

[jira] [Created] (SPARK-42504) NestedColumnAliasing support pruning adjacent projects

2023-02-20 Thread XiDuo You (Jira)
XiDuo You created SPARK-42504: - Summary: NestedColumnAliasing support pruning adjacent projects Key: SPARK-42504 URL: https://issues.apache.org/jira/browse/SPARK-42504 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42423) Add metadata column file block start and length

2023-02-13 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42423: -- Summary: Add metadata column file block start and length (was: Add metadata column file block start

[jira] [Updated] (SPARK-42423) Add metadata column file block start end

2023-02-13 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42423: -- Summary: Add metadata column file block start end (was: Add metadata column file block start) > Add

[jira] [Created] (SPARK-42423) Add metadata column file block start

2023-02-13 Thread XiDuo You (Jira)
XiDuo You created SPARK-42423: - Summary: Add metadata column file block start Key: SPARK-42423 URL: https://issues.apache.org/jira/browse/SPARK-42423 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37581) sql hang at planning stage

2023-02-05 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684376#comment-17684376 ] XiDuo You commented on SPARK-37581: --- This should be resovled by SPARK-38138. The root reason is dpp

[jira] [Comment Edited] (SPARK-41793) Incorrect result for window frames defined by a range clause on large decimals

2023-02-03 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683753#comment-17683753 ] XiDuo You edited comment on SPARK-41793 at 2/3/23 9:03 AM: --- I'm not sure this

[jira] [Commented] (SPARK-41793) Incorrect result for window frames defined by a range clause on large decimals

2023-02-03 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683753#comment-17683753 ] XiDuo You commented on SPARK-41793: --- I'm not sure this is a correctness bug but I think it's more like

[jira] [Created] (SPARK-42331) Fix metadata col can not been resolved

2023-02-02 Thread XiDuo You (Jira)
XiDuo You created SPARK-42331: - Summary: Fix metadata col can not been resolved Key: SPARK-42331 URL: https://issues.apache.org/jira/browse/SPARK-42331 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-35725) Support repartition expand partitions in AQE

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682473#comment-17682473 ] XiDuo You edited comment on SPARK-35725 at 1/31/23 10:08 AM: - [~Penglei Shi]

[jira] [Comment Edited] (SPARK-35725) Support repartition expand partitions in AQE

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682473#comment-17682473 ] XiDuo You edited comment on SPARK-35725 at 1/31/23 10:07 AM: - [~Penglei Shi]

[jira] [Commented] (SPARK-35725) Support repartition expand partitions in AQE

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682473#comment-17682473 ] XiDuo You commented on SPARK-35725: --- [~Penglei Shi] Kyuubi community has a Spark extension to support

[jira] [Commented] (SPARK-35725) Support repartition expand partitions in AQE

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682460#comment-17682460 ] XiDuo You commented on SPARK-35725: --- [~Penglei Shi] , It will cause inconsistent. The feature

[jira] [Updated] (SPARK-42251) Forbid deicmal type if precision less than 1

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42251: -- Summary: Forbid deicmal type if precision less than 1 (was: Forbid deicmal type if precision is 0)

[jira] [Updated] (SPARK-42251) Forbid deicmal type if precision is 0

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42251: -- Description: Spark does not actually support decimal type with 0 precision. e.g.   {code:java} –

[jira] [Updated] (SPARK-42251) Forbid deicmal type if precision is 0

2023-01-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42251: -- Summary: Forbid deicmal type if precision is 0 (was: Forbid deicmal(0, 0)) > Forbid deicmal type if

[jira] [Updated] (SPARK-42251) Forbid deicmal(0, 0)

2023-01-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42251: -- Description: Spark does not actually support decimal(0, 0). e.g.   {code:java} – work with in-memory

[jira] [Created] (SPARK-42251) Forbid deicmal(0, 0)

2023-01-30 Thread XiDuo You (Jira)
XiDuo You created SPARK-42251: - Summary: Forbid deicmal(0, 0) Key: SPARK-42251 URL: https://issues.apache.org/jira/browse/SPARK-42251 Project: Spark Issue Type: Improvement Components:

[jira] [Updated] (SPARK-42101) Wrap InMemoryTableScanExec with QueryStage

2023-01-17 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42101: -- Summary: Wrap InMemoryTableScanExec with QueryStage (was: Wrap InMemoryTableScanExec + AQE with

[jira] [Updated] (SPARK-42101) Wrap InMemoryTableScanExec with QueryStage

2023-01-17 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-42101: -- Description: The first access to the cached plan which is enable AQE is tricky. Currently, we can

[jira] [Created] (SPARK-42101) Wrap InMemoryTableScanExec + AQE with QueryStage

2023-01-17 Thread XiDuo You (Jira)
XiDuo You created SPARK-42101: - Summary: Wrap InMemoryTableScanExec + AQE with QueryStage Key: SPARK-42101 URL: https://issues.apache.org/jira/browse/SPARK-42101 Project: Spark Issue Type:

  1   2   3   4   5   >