[jira] [Updated] (SPARK-38853) optimizeSkewsInRebalancePartitions has performance issue

2022-04-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38853: Description: How to reproduce this issue: {code:sql} CREATE TABLE t USING PARQUET AS SELECT /*

[jira] [Created] (SPARK-38853) optimizeSkewsInRebalancePartitions has performance issue

2022-04-11 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38853: --- Summary: optimizeSkewsInRebalancePartitions has performance issue Key: SPARK-38853 URL: https://issues.apache.org/jira/browse/SPARK-38853 Project: Spark Issue

[jira] [Reopened] (SPARK-34444) Pushdown scalar-subquery filter to FileSourceScan

2022-04-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-3: - > Pushdown scalar-subquery filter to FileSourceScan > --

[jira] [Created] (SPARK-38843) Fix translate metadata col filters

2022-04-09 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38843: --- Summary: Fix translate metadata col filters Key: SPARK-38843 URL: https://issues.apache.org/jira/browse/SPARK-38843 Project: Spark Issue Type: Bug Co

[jira] [Resolved] (SPARK-38841) Enable Bloom filter join by default

2022-04-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38841. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36102 [https://gith

[jira] [Assigned] (SPARK-38841) Enable Bloom filter join by default

2022-04-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38841: --- Assignee: Yingyi Bu > Enable Bloom filter join by default > ---

[jira] [Resolved] (SPARK-38825) Add a test to cover parquet notIn filter

2022-04-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38825. - Fix Version/s: 3.3.0 Assignee: Huaxin Gao Resolution: Fixed > Add a test to cove

[jira] [Updated] (SPARK-38808) Windows, Spark 3.2.1: spark-shell command throwing this error: SparkContext: Error initializing SparkContext

2022-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38808: Target Version/s: (was: 3.2.1) > Windows, Spark 3.2.1: spark-shell command throwing this error:

[jira] [Assigned] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

2022-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38772: --- Assignee: Yuming Wang > Formatting the log plan in AdaptiveSparkPlanExec >

[jira] [Resolved] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

2022-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38772. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36045 [https://gith

[jira] [Created] (SPARK-38797) Runtime Filter support pushdown through window

2022-04-05 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38797: --- Summary: Runtime Filter support pushdown through window Key: SPARK-38797 URL: https://issues.apache.org/jira/browse/SPARK-38797 Project: Spark Issue Type: Impr

[jira] [Resolved] (SPARK-38778) Replace http with https for project url in pom

2022-04-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38778. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36053 [https://gith

[jira] [Assigned] (SPARK-38778) Replace http with https for project url in pom

2022-04-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38778: --- Assignee: Kent Yao > Replace http with https for project url in pom > -

[jira] [Resolved] (SPARK-38757) Update the Oracle docker image version used for test and integration

2022-04-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38757. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36036 [https://gith

[jira] [Assigned] (SPARK-38757) Update the Oracle docker image version used for test and integration

2022-04-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38757: --- Assignee: Luca Canali > Update the Oracle docker image version used for test and integratio

[jira] [Created] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

2022-04-02 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38772: --- Summary: Formatting the log plan in AdaptiveSparkPlanExec Key: SPARK-38772 URL: https://issues.apache.org/jira/browse/SPARK-38772 Project: Spark Issue Type: Im

[jira] [Updated] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

2022-04-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38771: Description: Insert a bloom filter on one side of the join if it may be spill when sorting and th

[jira] [Updated] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

2022-04-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38771: Attachment: Special case.png > Introduce adaptive Bloom filter Join to reduce spilling to disk whe

[jira] [Created] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

2022-04-02 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38771: --- Summary: Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin Key: SPARK-38771 URL: https://issues.apache.org/jira/browse/SPARK-38771

[jira] [Created] (SPARK-38708) Upgrade Hive Metastore Client to the 3.1.3 for Hive 3.1

2022-03-30 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38708: --- Summary: Upgrade Hive Metastore Client to the 3.1.3 for Hive 3.1 Key: SPARK-38708 URL: https://issues.apache.org/jira/browse/SPARK-38708 Project: Spark Issue T

[jira] [Commented] (SPARK-38703) High GC and memory footprint after switch to ZSTD

2022-03-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515003#comment-17515003 ] Yuming Wang commented on SPARK-38703: - How to reproduce this issue? > High GC and m

[jira] [Resolved] (SPARK-38702) Is there a spark (spark-sql_2.12) using log4j 2.17.X ?

2022-03-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38702. - Resolution: Invalid > Is there a spark (spark-sql_2.12) using log4j 2.17.X ? >

[jira] [Commented] (SPARK-38702) Is there a spark (spark-sql_2.12) using log4j 2.17.X ?

2022-03-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515001#comment-17515001 ] Yuming Wang commented on SPARK-38702: - Could you use snapshot: https://repository.a

[jira] [Commented] (SPARK-38695) ORC can not surport the dataType,such as char or varchar

2022-03-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514547#comment-17514547 ] Yuming Wang commented on SPARK-38695: - How to reproduce this issue? > ORC can not s

[jira] (SPARK-38674) Remove useless deduplicate in SubqueryBroadcastExec

2022-03-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38674 ] Yuming Wang deleted comment on SPARK-38674: - was (Author: q79969786): https://github.com/apache/spark/pull/35989 > Remove useless deduplicate in SubqueryBroadcastExec > -

[jira] [Commented] (SPARK-38674) Remove useless deduplicate in SubqueryBroadcastExec

2022-03-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513301#comment-17513301 ] Yuming Wang commented on SPARK-38674: - https://github.com/apache/spark/pull/35989 >

[jira] [Updated] (SPARK-38674) Remove useless deduplicate in SubqueryBroadcastExec

2022-03-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38674: Description: Distinct performance: https://github.com/apache/spark/pull/29642#discussion_r51160649

[jira] [Created] (SPARK-38674) Remove useless deduplicate in SubqueryBroadcastExec

2022-03-28 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38674: --- Summary: Remove useless deduplicate in SubqueryBroadcastExec Key: SPARK-38674 URL: https://issues.apache.org/jira/browse/SPARK-38674 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513142#comment-17513142 ] Yuming Wang commented on SPARK-38671: - https://github.com/apache/spark/pull/35986 >

[jira] [Created] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38671: --- Summary: Publish snapshot from branch-3.3 Key: SPARK-38671 URL: https://issues.apache.org/jira/browse/SPARK-38671 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-38665) upgrade jackson due to CVE-2020-36518

2022-03-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38665. - Fix Version/s: 3.3.0 Assignee: Cheng Pan Resolution: Fixed Issue resolved by pul

[jira] [Updated] (SPARK-38570) Incorrect DynamicPartitionPruning caused by Literal

2022-03-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38570: Fix Version/s: 3.1.4 (was: 3.1.3) > Incorrect DynamicPartitionPruning cause

[jira] [Updated] (SPARK-38570) Incorrect DynamicPartitionPruning caused by Literal

2022-03-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38570: Fix Version/s: 3.1.3 3.0.4 3.2.2 > Incorrect DynamicPartitio

[jira] [Updated] (SPARK-38663) Remove inaccessible repository: https://dl.bintray.com

2022-03-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38663: Summary: Remove inaccessible repository: https://dl.bintray.com (was: Could not connect to dl.bin

[jira] [Created] (SPARK-38663) Could not connect to dl.bintray.com

2022-03-25 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38663: --- Summary: Could not connect to dl.bintray.com Key: SPARK-38663 URL: https://issues.apache.org/jira/browse/SPARK-38663 Project: Spark Issue Type: Bug C

[jira] [Resolved] (SPARK-38570) Incorrect DynamicPartitionPruning caused by Literal

2022-03-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38570. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35878 https://githu

[jira] [Assigned] (SPARK-38570) Incorrect DynamicPartitionPruning caused by Literal

2022-03-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38570: --- Assignee: mcdull_zhang > Incorrect DynamicPartitionPruning caused by Literal >

[jira] [Resolved] (SPARK-38624) Reduce UnsafeProjection.create call times when Percentile function serializes the aggregation buffer object

2022-03-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38624. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35938 [https://gith

[jira] [Assigned] (SPARK-38624) Reduce UnsafeProjection.create call times when Percentile function serializes the aggregation buffer object

2022-03-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38624: --- Assignee: w l > Reduce UnsafeProjection.create call times when Percentile function serializ

[jira] [Created] (SPARK-38634) Upgrade hive-service-rpc to 4.0.0

2022-03-23 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38634: --- Summary: Upgrade hive-service-rpc to 4.0.0 Key: SPARK-38634 URL: https://issues.apache.org/jira/browse/SPARK-38634 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-38624) Reduce UnsafeProjection.create call times when Percentile function serializes the aggregation buffer object

2022-03-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38624: Fix Version/s: (was: 3.4.0) > Reduce UnsafeProjection.create call times when Percentile functi

[jira] [Resolved] (SPARK-38579) Requesting Restful API can cause NullPointerException

2022-03-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38579. - Fix Version/s: 3.3.0 3.2.2 3.1.3 Resolution: Fixed

[jira] [Assigned] (SPARK-38579) Requesting Restful API can cause NullPointerException

2022-03-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38579: --- Assignee: Yimin Yang > Requesting Restful API can cause NullPointerException >

[jira] [Updated] (SPARK-38568) Upgrade ZSTD-JNI to 1.5.2-2

2022-03-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38568: Reporter: Yuming Wang (was: Dongjoon Hyun) > Upgrade ZSTD-JNI to 1.5.2-2 > --

[jira] [Commented] (SPARK-38568) Upgrade ZSTD-JNI to 1.5.2-2

2022-03-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507942#comment-17507942 ] Yuming Wang commented on SPARK-38568: - OK > Upgrade ZSTD-JNI to 1.5.2-2 > -

[jira] [Created] (SPARK-38568) Upgrade ZSTD-JNI to 1.5.2-2

2022-03-16 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38568: --- Summary: Upgrade ZSTD-JNI to 1.5.2-2 Key: SPARK-38568 URL: https://issues.apache.org/jira/browse/SPARK-38568 Project: Spark Issue Type: Improvement C

[jira] [Updated] (SPARK-38568) Upgrade ZSTD-JNI to 1.5.2-2

2022-03-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38568: Fix Version/s: (was: 3.3.0) > Upgrade ZSTD-JNI to 1.5.2-2 > --- > >

[jira] [Assigned] (SPARK-38568) Upgrade ZSTD-JNI to 1.5.2-2

2022-03-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38568: --- Assignee: (was: Dongjoon Hyun) > Upgrade ZSTD-JNI to 1.5.2-2 >

[jira] [Updated] (SPARK-38544) Upgrade log4j2 to 2.17.2

2022-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38544: Fix Version/s: (was: 3.3.0) > Upgrade log4j2 to 2.17.2 > > >

[jira] [Reopened] (SPARK-38544) Upgrade log4j2 to 2.17.2

2022-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-38544: - > Upgrade log4j2 to 2.17.2 > > > Key: SPARK-38544 >

[jira] [Updated] (SPARK-36194) Add a logical plan visitor to propagate the distinct attributes

2022-03-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-36194: Summary: Add a logical plan visitor to propagate the distinct attributes (was: Remove the aggrega

[jira] [Resolved] (SPARK-36194) Remove the aggregation from left semi/anti join if the same aggregation has already been done on left side

2022-03-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-36194. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35779 [https://gith

[jira] [Assigned] (SPARK-36194) Remove the aggregation from left semi/anti join if the same aggregation has already been done on left side

2022-03-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-36194: --- Assignee: Yuming Wang > Remove the aggregation from left semi/anti join if the same aggrega

[jira] [Resolved] (SPARK-38540) Upgrade compress-lzf from 1.0.3 to 1.1.0

2022-03-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38540. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35834 [https://gith

[jira] [Assigned] (SPARK-38540) Upgrade compress-lzf from 1.0.3 to 1.1.0

2022-03-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38540: --- Assignee: Yang Jie > Upgrade compress-lzf from 1.0.3 to 1.1.0 > ---

[jira] [Assigned] (SPARK-38538) Fix driver environment verification in BasicDriverFeatureStepSuite

2022-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38538: --- Assignee: Dongjoon Hyun > Fix driver environment verification in BasicDriverFeatureStepSuit

[jira] [Resolved] (SPARK-38538) Fix driver environment verification in BasicDriverFeatureStepSuite

2022-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38538. - Fix Version/s: 3.3.0 3.0.4 3.2.2 3.1.3

[jira] [Updated] (SPARK-38516) Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided

2022-03-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38516: Summary: Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided (w

[jira] [Created] (SPARK-38516) Add log4j-core and log4j-api to classpath if active hadoop-provided

2022-03-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38516: --- Summary: Add log4j-core and log4j-api to classpath if active hadoop-provided Key: SPARK-38516 URL: https://issues.apache.org/jira/browse/SPARK-38516 Project: Spark

[jira] [Created] (SPARK-38506) Push partial aggregation through join

2022-03-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38506: --- Summary: Push partial aggregation through join Key: SPARK-38506 URL: https://issues.apache.org/jira/browse/SPARK-38506 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-38505) Make partial aggregation adaptive

2022-03-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38505: --- Summary: Make partial aggregation adaptive Key: SPARK-38505 URL: https://issues.apache.org/jira/browse/SPARK-38505 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Description: {code:scala} spark.range(200L).selectExpr("id AS a").createTempView("t1") spa

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Description: {code:scala} spark.range(200L).selectExpr("id AS a").createTempView("t1") spa

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Description: {code:scala} spark.range(200L).selectExpr("id AS a").createTempView("t1") spa

[jira] [Created] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38489: --- Summary: Aggregate.groupOnly support foldable expressions Key: SPARK-38489 URL: https://issues.apache.org/jira/browse/SPARK-38489 Project: Spark Issue Type: Im

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Fix Version/s: (was: 3.3.0) > Aggregate.groupOnly support foldable expressions > -

[jira] [Assigned] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38489: --- Assignee: (was: Yuming Wang) > Aggregate.groupOnly support foldable expressions > -

[jira] [Commented] (SPARK-38282) Avoid duplicating complex partitioning expressions

2022-03-07 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502674#comment-17502674 ] Yuming Wang commented on SPARK-38282: - Some related PRs: https://github.com/apache/s

[jira] [Commented] (SPARK-37989) Push down limit through Aggregate if it is group only

2022-01-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480963#comment-17480963 ] Yuming Wang commented on SPARK-37989: - Benchmark 5 {code:java} import org.apache.spa

[jira] [Commented] (SPARK-37989) Push down limit through Aggregate if it is group only

2022-01-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480892#comment-17480892 ] Yuming Wang commented on SPARK-37989: - Benchmark 4 {code:scala} import org.apache.s

[jira] [Commented] (SPARK-37989) Push down limit through Aggregate if it is group only

2022-01-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480882#comment-17480882 ] Yuming Wang commented on SPARK-37989: - Benchmark 3 {noformat} import org.apache.spar

[jira] [Commented] (SPARK-37989) Push down limit through Aggregate if it is group only

2022-01-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480855#comment-17480855 ] Yuming Wang commented on SPARK-37989: - Benchmark 2 {code:scala} import org.apache.s

[jira] [Commented] (SPARK-37989) Push down limit through Aggregate if it is group only

2022-01-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480839#comment-17480839 ] Yuming Wang commented on SPARK-37989: - Benchmark 1 {code:scala} import org.apache.s

[jira] [Created] (SPARK-37989) Push down limit through Aggregate if it is group only

2022-01-22 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37989: --- Summary: Push down limit through Aggregate if it is group only Key: SPARK-37989 URL: https://issues.apache.org/jira/browse/SPARK-37989 Project: Spark Issue Typ

[jira] [Updated] (SPARK-37922) Combine to one cast if we can safely up-cast two casts

2022-01-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37922: Summary: Combine to one cast if we can safely up-cast two casts (was: Remove useless cast) > Com

[jira] [Created] (SPARK-37922) Remove useless cast

2022-01-16 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37922: --- Summary: Remove useless cast Key: SPARK-37922 URL: https://issues.apache.org/jira/browse/SPARK-37922 Project: Spark Issue Type: Improvement Component

[jira] [Updated] (SPARK-37917) Push down limit 1 for right side of left semi/anti join

2022-01-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37917: Summary: Push down limit 1 for right side of left semi/anti join (was: Push down limit for right

[jira] [Created] (SPARK-37917) Push down limit for right side of left semi/anti join

2022-01-15 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37917: --- Summary: Push down limit for right side of left semi/anti join Key: SPARK-37917 URL: https://issues.apache.org/jira/browse/SPARK-37917 Project: Spark Issue Typ

[jira] [Assigned] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

2022-01-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34514: --- Assignee: Cheng Su (was: Apache Spark) > Push down limit for LEFT SEMI and LEFT ANTI join

[jira] [Created] (SPARK-37915) Push down deterministic projection through SQL UNION

2022-01-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37915: --- Summary: Push down deterministic projection through SQL UNION Key: SPARK-37915 URL: https://issues.apache.org/jira/browse/SPARK-37915 Project: Spark Issue Type

[jira] [Updated] (SPARK-37899) EliminateInnerJoin to support convert inner join to left semi join

2022-01-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37899: Summary: EliminateInnerJoin to support convert inner join to left semi join (was: EliminateInnerJ

[jira] [Created] (SPARK-37899) EliminateInnerJoin

2022-01-13 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37899: --- Summary: EliminateInnerJoin Key: SPARK-37899 URL: https://issues.apache.org/jira/browse/SPARK-37899 Project: Spark Issue Type: Improvement Components

[jira] [Resolved] (SPARK-37828) Push down filters through RebalancePartitions

2022-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37828. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35118 [https://gith

[jira] [Assigned] (SPARK-37828) Push down filters through RebalancePartitions

2022-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-37828: --- Assignee: Yuming Wang > Push down filters through RebalancePartitions > ---

[jira] [Assigned] (SPARK-37838) Upgrade scalatestplus artifacts to 3.3.0-SNAP3

2022-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-37838: --- Assignee: Yuming Wang > Upgrade scalatestplus artifacts to 3.3.0-SNAP3 > --

[jira] [Resolved] (SPARK-37838) Upgrade scalatestplus artifacts to 3.3.0-SNAP3

2022-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37838. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35128 [https://gith

[jira] [Created] (SPARK-37838) Upgrade scalatestplus artifacts to 3.3.0-SNAP3

2022-01-06 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37838: --- Summary: Upgrade scalatestplus artifacts to 3.3.0-SNAP3 Key: SPARK-37838 URL: https://issues.apache.org/jira/browse/SPARK-37838 Project: Spark Issue Type: Impr

[jira] [Created] (SPARK-37828) Push down filters through RebalancePartitions

2022-01-06 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37828: --- Summary: Push down filters through RebalancePartitions Key: SPARK-37828 URL: https://issues.apache.org/jira/browse/SPARK-37828 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-37807) Hivethriftserver2 send proper exception message

2022-01-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37807: Fix Version/s: (was: 3.3.0) > Hivethriftserver2 send proper exception message > --

[jira] [Created] (SPARK-37642) Fall back to SortMergeJoin if BroadcastHashJoin has OOM issue

2021-12-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37642: --- Summary: Fall back to SortMergeJoin if BroadcastHashJoin has OOM issue Key: SPARK-37642 URL: https://issues.apache.org/jira/browse/SPARK-37642 Project: Spark

[jira] [Commented] (SPARK-37626) Upgrade libthrift to 1.15.0

2021-12-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17458173#comment-17458173 ] Yuming Wang commented on SPARK-37626: - We need to upgrade to 0.16.0 because we need

[jira] [Resolved] (SPARK-37565) Upgrade mysql-connector-java to 8.0.27

2021-12-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37565. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34827 [https://gith

[jira] [Assigned] (SPARK-37565) Upgrade mysql-connector-java to 8.0.27

2021-12-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-37565: --- Assignee: Yuming Wang > Upgrade mysql-connector-java to 8.0.27 > --

[jira] [Commented] (SPARK-37609) Transient StackOverflowError on DataFrame from Catalyst QueryPlan

2021-12-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457481#comment-17457481 ] Yuming Wang commented on SPARK-37609: - How to reproduce this issue? > Transient Sta

[jira] [Commented] (SPARK-37603) org.apache.spark.sql.catalyst.expressions.ListQuery; no valid constructor

2021-12-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456920#comment-17456920 ] Yuming Wang commented on SPARK-37603: - How to reproduce this issue. > org.apache.sp

[jira] [Created] (SPARK-37565) Upgrade mysql-connector-java to 8.0.27

2021-12-06 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37565: --- Summary: Upgrade mysql-connector-java to 8.0.27 Key: SPARK-37565 URL: https://issues.apache.org/jira/browse/SPARK-37565 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-37549) Support set parallel through data source properties

2021-12-05 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37549: --- Summary: Support set parallel through data source properties Key: SPARK-37549 URL: https://issues.apache.org/jira/browse/SPARK-37549 Project: Spark Issue Type:

[jira] [Created] (SPARK-37535) Change default spark.io.compression.codec to zstd

2021-12-03 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-37535: --- Summary: Change default spark.io.compression.codec to zstd Key: SPARK-37535 URL: https://issues.apache.org/jira/browse/SPARK-37535 Project: Spark Issue Type: S

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-12-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452730#comment-17452730 ] Yuming Wang commented on SPARK-18105: - Workaround this issue by set spark.io.compres

[jira] [Resolved] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37270. - Fix Version/s: 3.2.1 Resolution: Fixed Issue resolved by pull request 34627 [https://gith

<    4   5   6   7   8   9   10   11   12   13   >