[jira] [Created] (SPARK-39291) Fetch blocks and open stream should not respond a closed channel

2022-05-25 Thread XiDuo You (Jira)
XiDuo You created SPARK-39291: - Summary: Fetch blocks and open stream should not respond a closed channel Key: SPARK-39291 URL: https://issues.apache.org/jira/browse/SPARK-39291 Project: Spark

[jira] [Created] (SPARK-39267) Clean up dsl unnecessary symbol

2022-05-23 Thread XiDuo You (Jira)
XiDuo You created SPARK-39267: - Summary: Clean up dsl unnecessary symbol Key: SPARK-39267 URL: https://issues.apache.org/jira/browse/SPARK-39267 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-39220) codegen cause NullPointException

2022-05-19 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539876#comment-17539876 ] XiDuo You commented on SPARK-39220: --- is it possible to also provide a stack log ? > codegen cause

[jira] [Updated] (SPARK-39172) Remove outer join if all output come from streamed side and buffered side keys exist unique key

2022-05-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-39172: -- Summary: Remove outer join if all output come from streamed side and buffered side keys exist unique

[jira] [Created] (SPARK-39172) Remove outer join if all output come from streamed side and buffered side keys exist unique

2022-05-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-39172: - Summary: Remove outer join if all output come from streamed side and buffered side keys exist unique Key: SPARK-39172 URL: https://issues.apache.org/jira/browse/SPARK-39172

[jira] [Commented] (SPARK-39104) Null Pointer Exeption on unpersist call

2022-05-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534136#comment-17534136 ] XiDuo You commented on SPARK-39104: --- it seems this bug also exists at 3.3.0 branch > Null Pointer

[jira] [Commented] (SPARK-39132) spark3.2.1 cache throw NPE

2022-05-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534134#comment-17534134 ] XiDuo You commented on SPARK-39132: --- same bug with SPARK-39104 > spark3.2.1 cache throw NPE >

[jira] [Created] (SPARK-39122) Python UDF does not follow the conditional expression evaluation order

2022-05-07 Thread XiDuo You (Jira)
XiDuo You created SPARK-39122: - Summary: Python UDF does not follow the conditional expression evaluation order Key: SPARK-39122 URL: https://issues.apache.org/jira/browse/SPARK-39122 Project: Spark

[jira] [Created] (SPARK-39106) Correct conditional expression constant folding

2022-05-05 Thread XiDuo You (Jira)
XiDuo You created SPARK-39106: - Summary: Correct conditional expression constant folding Key: SPARK-39106 URL: https://issues.apache.org/jira/browse/SPARK-39106 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39105) Add ConditionalExpression trait

2022-05-05 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-39105: -- Description: For developers, if a custom conditional like expression contains common sub expression

[jira] [Updated] (SPARK-39105) Add ConditionalExpression trait

2022-05-05 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-39105: -- Description: For develpers, if a custom conditional like expression contains common sub expression

[jira] [Updated] (SPARK-39105) Add ConditionalExpression trait

2022-05-05 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-39105: -- Description: For develpers, if a custom conditional like expression contains common sub expression

[jira] [Created] (SPARK-39105) Add ConditionalExpression trait

2022-05-05 Thread XiDuo You (Jira)
XiDuo You created SPARK-39105: - Summary: Add ConditionalExpression trait Key: SPARK-39105 URL: https://issues.apache.org/jira/browse/SPARK-39105 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-39040) Respect NaNvl in EquivalentExpressions for expression elimination

2022-04-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-39040: -- Description: For example the query will fail: {code:java} set spark.sql.ansi.enabled=true; set

[jira] [Created] (SPARK-39040) Respect NaNvl in EquivalentExpressions for expression elimination

2022-04-27 Thread XiDuo You (Jira)
XiDuo You created SPARK-39040: - Summary: Respect NaNvl in EquivalentExpressions for expression elimination Key: SPARK-39040 URL: https://issues.apache.org/jira/browse/SPARK-39040 Project: Spark

[jira] [Created] (SPARK-39039) Conditional expression evaluation ordering

2022-04-27 Thread XiDuo You (Jira)
XiDuo You created SPARK-39039: - Summary: Conditional expression evaluation ordering Key: SPARK-39039 URL: https://issues.apache.org/jira/browse/SPARK-39039 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-37528) Schedule Tasks By Input Size

2022-04-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37528: -- Description: In general, the larger input data size means longer running time. So ideally, we can

[jira] [Updated] (SPARK-37528) Schedule Tasks By Input Size

2022-04-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37528: -- Description: In general, the larger input data size means longer running time. So ideally, we can

[jira] [Updated] (SPARK-37528) Schedule Tasks By Input Size

2022-04-21 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37528: -- Description: In general, the larger input data size means longer running time. So ideally, we can

[jira] [Created] (SPARK-38962) Fix wrong computeStats at DataSourceV2Relation

2022-04-19 Thread XiDuo You (Jira)
XiDuo You created SPARK-38962: - Summary: Fix wrong computeStats at DataSourceV2Relation Key: SPARK-38962 URL: https://issues.apache.org/jira/browse/SPARK-38962 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38962) Fix wrong computeStats at DataSourceV2Relation

2022-04-19 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38962: -- Issue Type: Bug (was: Improvement) > Fix wrong computeStats at DataSourceV2Relation >

[jira] [Commented] (SPARK-38667) Optimizer generates error when using inner join along with sequence

2022-04-19 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524692#comment-17524692 ] XiDuo You commented on SPARK-38667: --- So you can add a config to avoid this issue set

[jira] [Commented] (SPARK-38667) Optimizer generates error when using inner join along with sequence

2022-04-19 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524690#comment-17524690 ] XiDuo You commented on SPARK-38667: --- it was introduced by SPARK-32295 and fixed by SPARK-37392 >

[jira] [Created] (SPARK-38932) Datasource v2 support report unique keys

2022-04-18 Thread XiDuo You (Jira)
XiDuo You created SPARK-38932: - Summary: Datasource v2 support report unique keys Key: SPARK-38932 URL: https://issues.apache.org/jira/browse/SPARK-38932 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-38895) Unify the AQE shuffle read canonicalized

2022-04-14 Thread XiDuo You (Jira)
XiDuo You created SPARK-38895: - Summary: Unify the AQE shuffle read canonicalized Key: SPARK-38895 URL: https://issues.apache.org/jira/browse/SPARK-38895 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-38887) Support switch inner join side for sort merge join

2022-04-13 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38887: -- Summary: Support switch inner join side for sort merge join (was: Support swtich inner join side for

[jira] [Created] (SPARK-38887) Support swtich inner join side for sort merge join

2022-04-13 Thread XiDuo You (Jira)
XiDuo You created SPARK-38887: - Summary: Support swtich inner join side for sort merge join Key: SPARK-38887 URL: https://issues.apache.org/jira/browse/SPARK-38887 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38886) Remove outer join if aggregate functions are duplicate agnostic on streamed side

2022-04-13 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38886: -- Description: If aggregate child is outer join, and the aggregate references are all coming from the

[jira] [Created] (SPARK-38886) Remove outer join if aggregate functions are duplicate agnostic on streamed side

2022-04-13 Thread XiDuo You (Jira)
XiDuo You created SPARK-38886: - Summary: Remove outer join if aggregate functions are duplicate agnostic on streamed side Key: SPARK-38886 URL: https://issues.apache.org/jira/browse/SPARK-38886 Project:

[jira] [Commented] (SPARK-38853) optimizeSkewsInRebalancePartitions has performance issue

2022-04-11 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17520368#comment-17520368 ] XiDuo You commented on SPARK-38853: --- Some issues might cause driver hang during optimizing skew :

[jira] [Updated] (SPARK-38832) Remove unnecessary distinct in aggregate expression by distinctKeys

2022-04-08 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38832: -- Description: We can remove the distinct in aggregate expression if the child distinct semantics is

[jira] [Created] (SPARK-38832) Remove unnecessary distinct in aggregate expression by distinctKeys

2022-04-08 Thread XiDuo You (Jira)
XiDuo You created SPARK-38832: - Summary: Remove unnecessary distinct in aggregate expression by distinctKeys Key: SPARK-38832 URL: https://issues.apache.org/jira/browse/SPARK-38832 Project: Spark

[jira] [Updated] (SPARK-38162) Optimize one row plan in normal and AQE Optimizer

2022-04-06 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Improvement) > Optimize one row plan in

[jira] [Created] (SPARK-38773) Correct the Union output partitioning and ordering

2022-04-02 Thread XiDuo You (Jira)
XiDuo You created SPARK-38773: - Summary: Correct the Union output partitioning and ordering Key: SPARK-38773 URL: https://issues.apache.org/jira/browse/SPARK-38773 Project: Spark Issue Type:

[jira] [Updated] (SPARK-37528) Schedule Tasks By Input Size

2022-04-01 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37528: -- Affects Version/s: 3.4.0 (was: 3.3.0) > Schedule Tasks By Input Size >

[jira] [Updated] (SPARK-37528) Schedule Tasks By Input Size

2022-04-01 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37528: -- Summary: Schedule Tasks By Input Size (was: Support reorder tasks during scheduling by shuffle

[jira] [Updated] (SPARK-37528) Support reorder tasks during scheduling by shuffle partition size in AQE

2022-04-01 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37528: -- Description: In general, the larger input data size means longer running time. So ideally, we can

[jira] [Updated] (SPARK-38697) Extend SparkSessionExtensions to inject rules into AQE Optimizer

2022-03-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38697: -- Description: Provide a entrance for developer to play their logical plan with runtime optimizer in

[jira] [Created] (SPARK-38697) Extend SparkSessionExtensions to inject rules into AQE Optimizer

2022-03-30 Thread XiDuo You (Jira)
XiDuo You created SPARK-38697: - Summary: Extend SparkSessionExtensions to inject rules into AQE Optimizer Key: SPARK-38697 URL: https://issues.apache.org/jira/browse/SPARK-38697 Project: Spark

[jira] [Updated] (SPARK-38578) Avoid unnecessary sort in FileFormatWriter if user has specified sort in AQE

2022-03-16 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38578: -- Summary: Avoid unnecessary sort in FileFormatWriter if user has specified sort in AQE (was: Avoid

[jira] [Updated] (SPARK-38578) Avoid unnecessary sort in FileFormatWriter if user has specified sort

2022-03-16 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38578: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Improvement) > Avoid unnecessary sort in

[jira] [Created] (SPARK-38578) Avoid unnecessary sort in FileFormatWriter if user has specified sort

2022-03-16 Thread XiDuo You (Jira)
XiDuo You created SPARK-38578: - Summary: Avoid unnecessary sort in FileFormatWriter if user has specified sort Key: SPARK-38578 URL: https://issues.apache.org/jira/browse/SPARK-38578 Project: Spark

[jira] [Updated] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform

2022-03-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37796: -- Priority: Major (was: Minor) > ByteArrayMethods arrayEquals should fast skip the check of aligning

[jira] [Updated] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2022-03-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36992: -- Priority: Major (was: Minor) > Improve byte array sort perf by unify getPrefix function of

[jira] [Updated] (SPARK-37037) Improve byte array sort by unify compareTo function of UTF8String and ByteArray

2022-03-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37037: -- Priority: Major (was: Minor) > Improve byte array sort by unify compareTo function of UTF8String and

[jira] [Commented] (SPARK-38536) Spark 3 can not read mixed format partitions

2022-03-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17506789#comment-17506789 ] XiDuo You commented on SPARK-38536: --- it should be fixed by SPARK-36197 ? > Spark 3 can not read

[jira] [Updated] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38519: -- Description: BroadcastExchangeExec will wrap fatal exception inside SparkFatalException and unwarp

[jira] [Updated] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38519: -- Description: BroadcastExchangeExec will wrap fatal exception in SparkFatalException and unwarp it 

[jira] [Updated] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38519: -- Description: BroadcastExchangeExec will wrap fatal exception in SparkFatalException and unwarp it in

[jira] [Created] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-38519: - Summary: AQE throw exception should respect SparkFatalException Key: SPARK-38519 URL: https://issues.apache.org/jira/browse/SPARK-38519 Project: Spark Issue Type:

[jira] [Created] (SPARK-38410) Support specify initial partition number for rebalance

2022-03-03 Thread XiDuo You (Jira)
XiDuo You created SPARK-38410: - Summary: Support specify initial partition number for rebalance Key: SPARK-38410 URL: https://issues.apache.org/jira/browse/SPARK-38410 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38406) Improve perfermance of ShufflePartitionsUtil createSkewPartitionSpecs

2022-03-03 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38406: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Improvement) > Improve perfermance of

[jira] [Updated] (SPARK-38406) Improve perfermance of ShufflePartitionsUtil createSkewPartitionSpecs

2022-03-03 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38406: -- Description: If shuffle is skewed with tens of thousands of map partitions and reduce partitions in

[jira] [Created] (SPARK-38406) Improve perfermance of ShufflePartitionsUtil createSkewPartitionSpecs

2022-03-03 Thread XiDuo You (Jira)
XiDuo You created SPARK-38406: - Summary: Improve perfermance of ShufflePartitionsUtil createSkewPartitionSpecs Key: SPARK-38406 URL: https://issues.apache.org/jira/browse/SPARK-38406 Project: Spark

[jira] [Updated] (SPARK-38401) Unify get preferred locations for shuffle in AQE

2022-03-03 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38401: -- Description: It has several issues with method `ShuffledRowRDD#getPreferredLocations`. * it does not

[jira] [Created] (SPARK-38401) Unify get preferred locations for shuffle in AQE

2022-03-02 Thread XiDuo You (Jira)
XiDuo You created SPARK-38401: - Summary: Unify get preferred locations for shuffle in AQE Key: SPARK-38401 URL: https://issues.apache.org/jira/browse/SPARK-38401 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38322) Support query stage show runtime statistics in formatted explain mode

2022-02-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38322: -- Description: The formatted explalin mode is the powerful explain mode to show the details of query

[jira] [Created] (SPARK-38322) Support query stage show runtime statistics in formatted explain mode

2022-02-24 Thread XiDuo You (Jira)
XiDuo You created SPARK-38322: - Summary: Support query stage show runtime statistics in formatted explain mode Key: SPARK-38322 URL: https://issues.apache.org/jira/browse/SPARK-38322 Project: Spark

[jira] [Commented] (SPARK-38172) Adaptive coalesce not working with df persist

2022-02-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497870#comment-17497870 ] XiDuo You commented on SPARK-38172: --- thanks [~Naveenmts]  for the confirming ! > Adaptive coalesce

[jira] [Resolved] (SPARK-38172) Adaptive coalesce not working with df persist

2022-02-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-38172. --- Resolution: Won't Fix > Adaptive coalesce not working with df persist >

[jira] [Updated] (SPARK-38232) Explain formatted does not collect subqueries under query stage in AQE

2022-02-16 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38232: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Bug) > Explain formatted does not collect

[jira] [Updated] (SPARK-38232) Explain formatted does not collect subqueries under query stage in AQE

2022-02-16 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38232: -- Description: ExplainUtils have not catched QueryStageExec during collecting subquries. So we can not

[jira] [Created] (SPARK-38232) Explain formatted does not collect subqueries under query stage in AQE

2022-02-16 Thread XiDuo You (Jira)
XiDuo You created SPARK-38232: - Summary: Explain formatted does not collect subqueries under query stage in AQE Key: SPARK-38232 URL: https://issues.apache.org/jira/browse/SPARK-38232 Project: Spark

[jira] [Commented] (SPARK-38172) Adaptive coalesce not working with df persist

2022-02-11 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490773#comment-17490773 ] XiDuo You commented on SPARK-38172: --- hi [~Naveenmts]  have you tried enable this config ? {code:java}

[jira] [Updated] (SPARK-38185) Fix data incorrect if aggregate function is empty

2022-02-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38185: -- Summary: Fix data incorrect if aggregate function is empty (was: Fix data incorrect if aggregate is

[jira] [Updated] (SPARK-38185) Fix data incorrect if aggregate is group only with empty function

2022-02-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38185: -- Description: The group only condition should check if the aggregate expression is empty. In

[jira] [Created] (SPARK-38185) Fix data incorrect if aggregate is group only with empty function

2022-02-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-38185: - Summary: Fix data incorrect if aggregate is group only with empty function Key: SPARK-38185 URL: https://issues.apache.org/jira/browse/SPARK-38185 Project: Spark

[jira] [Updated] (SPARK-38182) Fix NoSuchElementException if pushed filter does not contain any references

2022-02-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38182: -- Description: reproduce: {code:java} CREATE TABLE t (c1 int) USING PARQUET; SET

[jira] [Created] (SPARK-38182) Fix NoSuchElementException if pushed filter does not contain any references

2022-02-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-38182: - Summary: Fix NoSuchElementException if pushed filter does not contain any references Key: SPARK-38182 URL: https://issues.apache.org/jira/browse/SPARK-38182 Project: Spark

[jira] [Updated] (SPARK-38177) Fix wrong transformExpressions in Optimizer

2022-02-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38177: -- Description: `transformExpressions` can only traverse all expressions in this current query plan, so

[jira] [Created] (SPARK-38177) Fix wrong transformExpressions in Optimizer

2022-02-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-38177: - Summary: Fix wrong transformExpressions in Optimizer Key: SPARK-38177 URL: https://issues.apache.org/jira/browse/SPARK-38177 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38162) Optimize one row plan in normal and AQE Optimizer

2022-02-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Description: Optimize the plan if its max row is equal to or less than 1 in these cases: - if the

[jira] [Updated] (SPARK-38162) Optimize one row plan in normal and AQE Optimizer

2022-02-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Summary: Optimize one row plan in normal and AQE Optimizer (was: Optimize one max row plan in normal

[jira] [Updated] (SPARK-38162) Optimize one max row plan in normal and AQE Optimizer

2022-02-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Description: Optimize the plan if its max row is equal to or less than 1 in these cases: * if sort

[jira] [Updated] (SPARK-38162) Optimize one max row plan in normal and AQE Optimizer

2022-02-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Summary: Optimize one max row plan in normal and AQE Optimizer (was: Remove distinct in aggregate if

[jira] [Created] (SPARK-38162) Remove distinct in aggregate if its child is empty

2022-02-09 Thread XiDuo You (Jira)
XiDuo You created SPARK-38162: - Summary: Remove distinct in aggregate if its child is empty Key: SPARK-38162 URL: https://issues.apache.org/jira/browse/SPARK-38162 Project: Spark Issue Type:

[jira] [Created] (SPARK-38148) Do not add dynamic partition pruning if there exists static partition pruning

2022-02-08 Thread XiDuo You (Jira)
XiDuo You created SPARK-38148: - Summary: Do not add dynamic partition pruning if there exists static partition pruning Key: SPARK-38148 URL: https://issues.apache.org/jira/browse/SPARK-38148 Project:

[jira] [Commented] (SPARK-33832) Add an option in AQE to mitigate skew even if it causes an new shuffle

2022-01-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17483604#comment-17483604 ] XiDuo You commented on SPARK-33832: --- thank you [~dongjoon] ! > Add an option in AQE to mitigate skew

[jira] [Updated] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

2022-01-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Task) > AQE can change bhj to smj if no extra

[jira] [Commented] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

2022-01-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17483598#comment-17483598 ] XiDuo You commented on SPARK-38013: --- Add a test to cover this behavior > AQE can change bhj to smj if

[jira] [Resolved] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

2022-01-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-38013. --- Resolution: Won't Fix > AQE can change bhj to smj if no extra shuffle introduce >

[jira] [Commented] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

2022-01-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482209#comment-17482209 ] XiDuo You commented on SPARK-38013: --- seems it is allowed in AQE, not a bug otherwise .. > AQE can

[jira] [Updated] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

2022-01-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Issue Type: Task (was: Bug) > AQE can change bhj to smj if no extra shuffle introduce >

[jira] [Updated] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

2022-01-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Summary: AQE can change bhj to smj if no extra shuffle introduce (was: Fix AQE can change bhj to smj

[jira] [Updated] (SPARK-38013) Fix AQE can change bhj to smj if no extra shuffle introduce

2022-01-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Description: An example to reproduce the bug. {code:java} create table t1 as select 1 c1, 2 c2;

[jira] [Updated] (SPARK-38013) Fix AQE can change bhj to smj if no extra shuffle introduce

2022-01-25 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Description: An example to reproduce the bug. {code:java} create table t1 as select 1 c1, 2 c2;

[jira] [Updated] (SPARK-38013) Fix AQE can change bhj to smj if no extra shuffle introduce

2022-01-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Summary: Fix AQE can change bhj to smj if no extra shuffle introduce (was: Fix AQE can change bhj to

[jira] [Updated] (SPARK-38013) Fix AQE can change bhj to smj if the build side is aggregate

2022-01-24 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38013: -- Description: (was: The condition of isBroadcastStage is not completed. It does not catch the plan

[jira] [Created] (SPARK-38013) Fix AQE can change bhj to smj if the build side is aggregate

2022-01-24 Thread XiDuo You (Jira)
XiDuo You created SPARK-38013: - Summary: Fix AQE can change bhj to smj if the build side is aggregate Key: SPARK-38013 URL: https://issues.apache.org/jira/browse/SPARK-38013 Project: Spark

[jira] [Created] (SPARK-37949) Improve Rebalance statistics estimation

2022-01-17 Thread XiDuo You (Jira)
XiDuo You created SPARK-37949: - Summary: Improve Rebalance statistics estimation Key: SPARK-37949 URL: https://issues.apache.org/jira/browse/SPARK-37949 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-37904) Improve RebalancePartitions in rules of Optimizer

2022-01-13 Thread XiDuo You (Jira)
XiDuo You created SPARK-37904: - Summary: Improve RebalancePartitions in rules of Optimizer Key: SPARK-37904 URL: https://issues.apache.org/jira/browse/SPARK-37904 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37855) IllegalStateException when transforming an array inside a nested struct

2022-01-11 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472938#comment-17472938 ] XiDuo You commented on SPARK-37855: --- The regression seems from SPARK-35636, for the quick work around,

[jira] [Created] (SPARK-37862) RecordBinaryComparator should fast skip the check of aligning with unaligned platform

2022-01-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-37862: - Summary: RecordBinaryComparator should fast skip the check of aligning with unaligned platform Key: SPARK-37862 URL: https://issues.apache.org/jira/browse/SPARK-37862

[jira] [Updated] (SPARK-35442) Support propagate empty relation through aggregate

2022-01-07 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-35442: -- Summary: Support propagate empty relation through aggregate (was: Eliminate unnecessary join through

[jira] [Updated] (SPARK-35442) Eliminate unnecessary join through Aggregate

2022-01-07 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-35442: -- Description: The Aggregate in AQE is different with others, the `LogicalQueryStage` looks like

[jira] [Updated] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform

2021-12-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37796: -- Description: The method `arrayEquals` in `ByteArrayMethods` is critical function which is used in

[jira] [Updated] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform

2021-12-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37796: -- Summary: ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform

[jira] [Created] (SPARK-37796) ByteArrayMethods arrayEquals should fast skip the checking of aligned in unaligned platform

2021-12-30 Thread XiDuo You (Jira)
XiDuo You created SPARK-37796: - Summary: ByteArrayMethods arrayEquals should fast skip the checking of aligned in unaligned platform Key: SPARK-37796 URL: https://issues.apache.org/jira/browse/SPARK-37796

[jira] [Created] (SPARK-37659) Fix FsHistoryProvider race condition between list and delet log info

2021-12-15 Thread XiDuo You (Jira)
XiDuo You created SPARK-37659: - Summary: Fix FsHistoryProvider race condition between list and delet log info Key: SPARK-37659 URL: https://issues.apache.org/jira/browse/SPARK-37659 Project: Spark

[jira] [Created] (SPARK-37559) ShuffledRowRDD get preferred locations order by reduce size

2021-12-06 Thread XiDuo You (Jira)
XiDuo You created SPARK-37559: - Summary: ShuffledRowRDD get preferred locations order by reduce size Key: SPARK-37559 URL: https://issues.apache.org/jira/browse/SPARK-37559 Project: Spark Issue

<    1   2   3   4   5   >