[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: SPIP doc:

[jira] [Comment Edited] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839497#comment-17839497 ] Ke Jia edited comment on SPARK-47773 at 4/22/24 6:22 AM: - We have refined the

[jira] [Comment Edited] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839497#comment-17839497 ] Ke Jia edited comment on SPARK-47773 at 4/22/24 6:22 AM: - We have refined the

[jira] [Commented] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839497#comment-17839497 ] Ke Jia commented on SPARK-47773: We have refined the above

[jira] [Commented] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-09 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835171#comment-17835171 ] Ke Jia commented on SPARK-47773: [~viirya] Great, I have added comment permissions in the SPIP Doc. I'm

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: SPIP doc:

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: This

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: This

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: This

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: This

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-47773: --- Description: This SPIP outlines the integration of Gluten's physical plan conversion, validation, and

[jira] [Created] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-08 Thread Ke Jia (Jira)
Ke Jia created SPARK-47773: -- Summary: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines Key: SPARK-47773 URL: https://issues.apache.org/jira/browse/SPARK-47773

[jira] [Updated] (SPARK-43814) Spark cannot construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API

2023-05-31 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43814: --- Description:   When constructing the DecimalType in CatalystTypeConverters.convertToCatalyst(), spark

[jira] [Updated] (SPARK-43814) Spark cannot construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API

2023-05-31 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43814: --- Summary: Spark cannot construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API (was:

[jira] [Updated] (SPARK-43814) Spark cannot use the df.collect() result to construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API

2023-05-26 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43814: --- Description:   When using the df.collect() result to construct the DecimalType in

[jira] [Updated] (SPARK-43814) Spark cannot use the df.collect() result to construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API

2023-05-26 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43814: --- Description: When using the df.collect() result to construct the DecimalType in  Decimal scale (18) cannot

[jira] [Created] (SPARK-43814) Spark cannot use the df.collect() result to construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API

2023-05-26 Thread Ke Jia (Jira)
Ke Jia created SPARK-43814: -- Summary: Spark cannot use the df.collect() result to construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API Key: SPARK-43814 URL:

[jira] [Updated] (SPARK-43240) df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow]

2023-04-23 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43240: --- Affects Version/s: 3.3.2 (was: 3.2.2) > df.describe() method may- return wrong

[jira] [Updated] (SPARK-43240) df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow]

2023-04-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43240: --- Description: When calling the df.describe() method, the result  maybe wrong when the last RDD is

[jira] [Created] (SPARK-43240) df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow]

2023-04-22 Thread Ke Jia (Jira)
Ke Jia created SPARK-43240: -- Summary: df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow] Key: SPARK-43240 URL: https://issues.apache.org/jira/browse/SPARK-43240 Project: Spark

[jira] [Created] (SPARK-36898) Make the shuffle hash join factor configurable

2021-09-29 Thread Ke Jia (Jira)
Ke Jia created SPARK-36898: -- Summary: Make the shuffle hash join factor configurable Key: SPARK-36898 URL: https://issues.apache.org/jira/browse/SPARK-36898 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-35710) Support DPP + AQE when no reused broadcast exchange

2021-06-10 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-35710: --- Description: Support DPP + AQE when no reused broadcast exchange. > Support DPP + AQE when no reused

[jira] [Created] (SPARK-35710) Support DPP + AQE when no reused broadcast exchange

2021-06-10 Thread Ke Jia (Jira)
Ke Jia created SPARK-35710: -- Summary: Support DPP + AQE when no reused broadcast exchange Key: SPARK-35710 URL: https://issues.apache.org/jira/browse/SPARK-35710 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34637) Support DPP in AQE when the boradcast exchange can reused

2021-03-04 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-34637: --- Description: We have supported DPP in AQE when the join is Broadcast hash join before applying the AQE

[jira] [Updated] (SPARK-34637) Support DPP in AQE when the boradcast exchange can be reused

2021-03-04 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-34637: --- Summary: Support DPP in AQE when the boradcast exchange can be reused (was: Support DPP in AQE when the

[jira] [Created] (SPARK-34637) Support DPP in AQE when the boradcast exchange can reused

2021-03-04 Thread Ke Jia (Jira)
Ke Jia created SPARK-34637: -- Summary: Support DPP in AQE when the boradcast exchange can reused Key: SPARK-34637 URL: https://issues.apache.org/jira/browse/SPARK-34637 Project: Spark Issue Type:

[jira] [Created] (SPARK-34168) Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules

2021-01-19 Thread Ke Jia (Jira)
Ke Jia created SPARK-34168: -- Summary: Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules Key: SPARK-34168 URL: https://issues.apache.org/jira/browse/SPARK-34168 Project:

[jira] [Created] (SPARK-31524) Add metric to the split number for skew partition when enable AQE

2020-04-22 Thread Ke Jia (Jira)
Ke Jia created SPARK-31524: -- Summary: Add metric to the split number for skew partition when enable AQE Key: SPARK-31524 URL: https://issues.apache.org/jira/browse/SPARK-31524 Project: Spark

[jira] [Created] (SPARK-30922) Remove the max split config after changing the multi sub joins to multi sub partitions

2020-02-21 Thread Ke Jia (Jira)
Ke Jia created SPARK-30922: -- Summary: Remove the max split config after changing the multi sub joins to multi sub partitions Key: SPARK-30922 URL: https://issues.apache.org/jira/browse/SPARK-30922 Project:

[jira] [Created] (SPARK-30864) Add the user guide for Adaptive Query Execution

2020-02-17 Thread Ke Jia (Jira)
Ke Jia created SPARK-30864: -- Summary: Add the user guide for Adaptive Query Execution Key: SPARK-30864 URL: https://issues.apache.org/jira/browse/SPARK-30864 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-30188) Fix tests when enable Adaptive Query Execution

2020-02-04 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30188: --- Description: Fix the failed unit tests when enable Adaptive Query Execution. (was: Enable Adaptive Query

[jira] [Updated] (SPARK-30549) Fix the subquery metrics showing issue in UI When enable AQE

2020-01-17 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30549: --- Summary: Fix the subquery metrics showing issue in UI When enable AQE (was: Fix the subquery metrics

[jira] [Updated] (SPARK-30549) Fix the subquery metrics showing issue in UI When enable AQE

2020-01-17 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30549: --- Description: After merged [https://github.com/apache/spark/pull/25316], the subquery metrics can not be

[jira] [Created] (SPARK-30549) Fix the subquery metrics showing issue in UI

2020-01-17 Thread Ke Jia (Jira)
Ke Jia created SPARK-30549: -- Summary: Fix the subquery metrics showing issue in UI Key: SPARK-30549 URL: https://issues.apache.org/jira/browse/SPARK-30549 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-30524) Disable OptimizeSkewJoin rule if introducing additional shuffle.

2020-01-15 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30524: --- Description: The OptimizeSkewedJoin will break the outputPartitioning of origin SMJ. And it may introduce

[jira] [Created] (SPARK-30524) Disable OptimizeSkewJoin rule if introducing additional shuffle.

2020-01-15 Thread Ke Jia (Jira)
Ke Jia created SPARK-30524: -- Summary: Disable OptimizeSkewJoin rule if introducing additional shuffle. Key: SPARK-30524 URL: https://issues.apache.org/jira/browse/SPARK-30524 Project: Spark Issue

[jira] [Updated] (SPARK-30407) reset the metrics info of AdaptiveSparkPlanExec plan when enable aqe

2020-01-02 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30407: --- Description: Working on [https://github.com/apache/spark/pull/26813]. With on AQE, the metric info of

[jira] [Created] (SPARK-30407) reset the metrics info of AdaptiveSparkPlanExec plan when enable aqe

2020-01-02 Thread Ke Jia (Jira)
Ke Jia created SPARK-30407: -- Summary: reset the metrics info of AdaptiveSparkPlanExec plan when enable aqe Key: SPARK-30407 URL: https://issues.apache.org/jira/browse/SPARK-30407 Project: Spark

[jira] [Updated] (SPARK-30403) Fix the NoSuchElementException exception when enable AQE with InSubquery use case

2020-01-01 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30403: --- Description: After merged [https://github.com/apache/spark/pull/25854], we also need to handle the

[jira] [Updated] (SPARK-30403) Fix the NoSuchElementException exception when enable AQE with InSubquery use case

2020-01-01 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30403: --- Description: After merged [link title|[https://github.com/apache/spark/pull/25854]], we also need to handle

[jira] [Updated] (SPARK-30403) Fix the NoSuchElementException exception when enable AQE with InSubquery use case

2020-01-01 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-30403: --- Description: After merged [PR25854|[https://github.com/apache/spark/pull/25854]], we also need to handle

[jira] [Created] (SPARK-30403) Fix the NoSuchElementException exception when enable AQE with InSubquery use case

2020-01-01 Thread Ke Jia (Jira)
Ke Jia created SPARK-30403: -- Summary: Fix the NoSuchElementException exception when enable AQE with InSubquery use case Key: SPARK-30403 URL: https://issues.apache.org/jira/browse/SPARK-30403 Project: Spark

[jira] [Created] (SPARK-30291) Catch the exception when do materialize in AQE

2019-12-17 Thread Ke Jia (Jira)
Ke Jia created SPARK-30291: -- Summary: Catch the exception when do materialize in AQE Key: SPARK-30291 URL: https://issues.apache.org/jira/browse/SPARK-30291 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30232) Fix the the ArthmeticException by zero when enable AQE

2019-12-12 Thread Ke Jia (Jira)
Ke Jia created SPARK-30232: -- Summary: Fix the the ArthmeticException by zero when enable AQE Key: SPARK-30232 URL: https://issues.apache.org/jira/browse/SPARK-30232 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30213) Remove the mutable status in QueryStage when enable AQE

2019-12-10 Thread Ke Jia (Jira)
Ke Jia created SPARK-30213: -- Summary: Remove the mutable status in QueryStage when enable AQE Key: SPARK-30213 URL: https://issues.apache.org/jira/browse/SPARK-30213 Project: Spark Issue Type: New

[jira] [Created] (SPARK-30188) Enable Adaptive Query Execution default

2019-12-09 Thread Ke Jia (Jira)
Ke Jia created SPARK-30188: -- Summary: Enable Adaptive Query Execution default Key: SPARK-30188 URL: https://issues.apache.org/jira/browse/SPARK-30188 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-29954) collect the runtime statistics of row count in map stage

2019-12-01 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985853#comment-16985853 ] Ke Jia commented on SPARK-29954: [~hyukjin.kwon] Add the Jira description. Thanks. > collect the

[jira] [Updated] (SPARK-29954) collect the runtime statistics of row count in map stage

2019-12-01 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-29954: --- Description: We need the row count info to more accurately estimate the data skew situation when too many

[jira] [Created] (SPARK-29954) collect the runtime statistics of row count in map stage

2019-11-18 Thread Ke Jia (Jira)
Ke Jia created SPARK-29954: -- Summary: collect the runtime statistics of row count in map stage Key: SPARK-29954 URL: https://issues.apache.org/jira/browse/SPARK-29954 Project: Spark Issue Type:

[jira] [Created] (SPARK-29893) Improve the local reader performance by changing the task number from 1 to multi

2019-11-14 Thread Ke Jia (Jira)
Ke Jia created SPARK-29893: -- Summary: Improve the local reader performance by changing the task number from 1 to multi Key: SPARK-29893 URL: https://issues.apache.org/jira/browse/SPARK-29893 Project: Spark

[jira] [Updated] (SPARK-29792) SQL metrics cannot be updated to subqueries in AQE

2019-11-11 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-29792: --- Description: After merged  [SPARK-28583|https://issues.apache.org/jira/browse/SPARK-28583], the subqueries

[jira] [Updated] (SPARK-29552) Fix the flaky test failed in AdaptiveQueryExecSuite # multiple joins

2019-10-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-29552: --- Description: AQE will optimize the logical plan once there is query stage finished. So for inner join, when

[jira] [Created] (SPARK-29552) Fix the flaky test failed in AdaptiveQueryExecSuite # multiple joins

2019-10-22 Thread Ke Jia (Jira)
Ke Jia created SPARK-29552: -- Summary: Fix the flaky test failed in AdaptiveQueryExecSuite # multiple joins Key: SPARK-29552 URL: https://issues.apache.org/jira/browse/SPARK-29552 Project: Spark

[jira] [Commented] (SPARK-29544) Optimize skewed join at runtime with new Adaptive Execution

2019-10-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956744#comment-16956744 ] Ke Jia commented on SPARK-29544: [~wenchen] The attachment  is the design doc of skew join optimization.

[jira] [Updated] (SPARK-29544) Optimize skewed join at runtime with new Adaptive Execution

2019-10-22 Thread Ke Jia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-29544: --- Attachment: Skewed Join Optimization Design Doc.docx > Optimize skewed join at runtime with new Adaptive

[jira] [Created] (SPARK-29544) Optimize skewed join at runtime with new Adaptive Execution

2019-10-22 Thread Ke Jia (Jira)
Ke Jia created SPARK-29544: -- Summary: Optimize skewed join at runtime with new Adaptive Execution Key: SPARK-29544 URL: https://issues.apache.org/jira/browse/SPARK-29544 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28576) Fix the dead lock issue when enable new adaptive execution

2019-07-30 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28576: --- Description: After enable AE(SPARK-23128), we found the dead lock issue in Q6 1TB TPC-DS. The root cause is

[jira] [Updated] (SPARK-28576) Fix the dead lock issue when enable new adaptive execution

2019-07-30 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28576: --- Description: After enable AE([lSPARK-23128|https://issues.apache.org/jira/browse/SPARK-23128]), we found

[jira] [Updated] (SPARK-28576) Fix the dead lock issue when enable new adaptive execution

2019-07-30 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28576: --- Attachment: q6.sql physical plan.txt jstack.log > Fix the dead lock issue

[jira] [Created] (SPARK-28576) Fix the dead lock issue when enable new adaptive execution

2019-07-30 Thread Ke Jia (JIRA)
Ke Jia created SPARK-28576: -- Summary: Fix the dead lock issue when enable new adaptive execution Key: SPARK-28576 URL: https://issues.apache.org/jira/browse/SPARK-28576 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-07-29 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28560: --- Description: Implement a rule in the new adaptive execution framework introduced in SPARK-23128. This rule

[jira] [Created] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-07-29 Thread Ke Jia (JIRA)
Ke Jia created SPARK-28560: -- Summary: Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution Key: SPARK-28560 URL: https://issues.apache.org/jira/browse/SPARK-28560

[jira] [Updated] (SPARK-28046) OOM caused by building hash table when the compressed ratio of small table is normal

2019-06-13 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28046: --- Attachment: image-2019-06-14-10-34-53-379.png > OOM caused by building hash table when the compressed ratio

[jira] [Updated] (SPARK-28046) OOM caused by building hash table when the compressed ratio of small table is normal

2019-06-13 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28046: --- Description: Currently, spark will convert the sort merge join to broadcast hash join when the small table

[jira] [Created] (SPARK-28046) OOM caused by building hash table when the compressed ratio of small table is normal

2019-06-13 Thread Ke Jia (JIRA)
Ke Jia created SPARK-28046: -- Summary: OOM caused by building hash table when the compressed ratio of small table is normal Key: SPARK-28046 URL: https://issues.apache.org/jira/browse/SPARK-28046 Project:

[jira] [Commented] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-17 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744791#comment-16744791 ] Ke Jia commented on SPARK-26639: [~mgaido] Yes,  the current master also have above phenomenon. > The

[jira] [Commented] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-16 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744620#comment-16744620 ] Ke Jia commented on SPARK-26639: [~hyukjin.kwon] Thanks for your interesting.  As discussion in

[jira] [Commented] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-16 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744606#comment-16744606 ] Ke Jia commented on SPARK-26639: [@davies|https://github.com/davies] 

[jira] [Updated] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-16 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26639: --- Description: The subquery reuse feature has done in  [https://github.com/apache/spark/pull/14548] In my

[jira] [Updated] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-16 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26639: --- Description: The subquery reuse feature has done in 

[jira] [Updated] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-16 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26639: --- Description: The subquery reuse feature has done in 

[jira] [Created] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2019-01-16 Thread Ke Jia (JIRA)
Ke Jia created SPARK-26639: -- Summary: The reuse subquery function maybe does not work in SPARK SQL Key: SPARK-26639 URL: https://issues.apache.org/jira/browse/SPARK-26639 Project: Spark Issue

[jira] [Comment Edited] (SPARK-16958) Reuse subqueries within single query

2019-01-15 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743720#comment-16743720 ] Ke Jia edited comment on SPARK-16958 at 1/16/19 7:39 AM: - [~davies] hi, I left

[jira] [Commented] (SPARK-16958) Reuse subqueries within single query

2019-01-15 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743720#comment-16743720 ] Ke Jia commented on SPARK-16958: [~davies] hi, I left some comments in the

[jira] [Updated] (SPARK-26316) Because of the perf degradation in TPC-DS, we currently partial revert SPARK-21052:Add hash map metrics to join,

2018-12-09 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26316: --- Description: The code of  

[jira] [Created] (SPARK-26316) Because of the perf degradation in TPC-DS, we currently partial revert SPARK-21052:Add hash map metrics to join,

2018-12-09 Thread Ke Jia (JIRA)
Ke Jia created SPARK-26316: -- Summary: Because of the perf degradation in TPC-DS, we currently partial revert SPARK-21052:Add hash map metrics to join, Key: SPARK-26316 URL:

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-12-08 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713847#comment-16713847 ] Ke Jia commented on SPARK-26155: Upload the result of all queries in tpcds in 1TB data scale. > Spark

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-12-08 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Attachment: tpcds.result.xlsx > Spark SQL performance degradation after apply SPARK-21052 with Q19 of

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-12-03 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708223#comment-16708223 ] Ke Jia commented on SPARK-26155: [~cloud_fan] [~viirya]  Spark2.3 with the optimized patch can have the

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-12-03 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706837#comment-16706837 ] Ke Jia commented on SPARK-26155: [~cloud_fan]  sorry for the delay. The  revert PR is 

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-28 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701533#comment-16701533 ] Ke Jia commented on SPARK-26155: [~viirya] [~adrian-wang]  Here is the result in  spark2.1, spark2.3

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-27 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701317#comment-16701317 ] Ke Jia commented on SPARK-26155: [~viirya] Thanks for your reply.  > "Q19 analysis in Spark2.3 without

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-27 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Attachment: (was: Q19 analysis in Spark2.3 without L486 & 487.pdf) > Spark SQL performance degradation

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-27 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Attachment: Q19 analysis in Spark2.3 without L486&487.pdf > Spark SQL performance degradation after apply

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-23 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Attachment: q19.sql Q19 analysis in Spark2.3 without L486 & 487.pdf Q19

[jira] [Comment Edited] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-23 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696476#comment-16696476 ] Ke Jia edited comment on SPARK-26155 at 11/23/18 7:58 AM: -- *Cluster info:* | 

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-22 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696476#comment-16696476 ] Ke Jia commented on SPARK-26155: *Cluster info:* | |*Master Node*|*Worker Nodes* | |*Node*|1x |7x|

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-22 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Summary: Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale (was:

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 2.6TB scale

2018-11-22 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Description: In our test environment, we found a serious performance degradation issue in Spark2.3 when

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 2.6TB scale

2018-11-22 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Description: In our test environment, we found a serious performance degradation issue in Spark2.3 when

[jira] [Updated] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 2.6TB scale

2018-11-22 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-26155: --- Description: In our test environment, we found a serious performance degradation issue in Spark2.3 when

[jira] [Created] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 2.6TB scale

2018-11-22 Thread Ke Jia (JIRA)
Ke Jia created SPARK-26155: -- Summary: Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 2.6TB scale Key: SPARK-26155 URL: https://issues.apache.org/jira/browse/SPARK-26155

[jira] [Updated] (SPARK-8321) Authorization Support(on all operations not only DDL) in Spark Sql

2016-03-29 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-8321: -- Attachment: SparkSQLauthorizationDesignDocument.pdf > Authorization Support(on all operations not only DDL) in

[jira] [Updated] (SPARK-8321) Authorization Support(on all operations not only DDL) in Spark Sql

2015-12-17 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-8321: -- Attachment: SparkSQLauthorizationDesignDocument.pdf > Authorization Support(on all operations not only DDL) in