[jira] [Issue Comment Deleted] (SPARK-34931) CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.

2021-04-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34931: Comment: was deleted (was: User 'sarutak' has created a pull request for this issue:

[jira] [Updated] (SPARK-34920) Introduce SQLSTATE and ERRORCODE to SQL Exception

2021-03-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34920: Description: SQLSTATE is SQL standard state. Please see was: SQLSTATE is SQL standard state

[jira] [Created] (SPARK-34920) Introduce SQLSTATE and ERRORCODE to SQL Exception

2021-03-31 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34920: --- Summary: Introduce SQLSTATE and ERRORCODE to SQL Exception Key: SPARK-34920 URL: https://issues.apache.org/jira/browse/SPARK-34920 Project: Spark Issue Type:

[jira] [Commented] (SPARK-34897) The given data schema has less fields than the actual ORC physical schema

2021-03-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310500#comment-17310500 ] Yuming Wang commented on SPARK-34897: - We can workaround this issue by setting

[jira] [Updated] (SPARK-34897) The given data schema has less fields than the actual ORC physical schema

2021-03-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34897: Description: How to reproduce this issue: {code:scala} spark.sql( """ |CREATE TABLE `t1` (

[jira] [Created] (SPARK-34897) The given data schema has less fields than the actual ORC physical schema

2021-03-29 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34897: --- Summary: The given data schema has less fields than the actual ORC physical schema Key: SPARK-34897 URL: https://issues.apache.org/jira/browse/SPARK-34897 Project:

[jira] [Updated] (SPARK-33747) Avoid calling unregisterMapOutput when the map stage is being rerunning.

2021-03-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33747: Fix Version/s: (was: 3.0.1) (was: 2.4.5) > Avoid calling

[jira] [Created] (SPARK-34884) Improve dynamic partition pruning evaluation

2021-03-28 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34884: --- Summary: Improve dynamic partition pruning evaluation Key: SPARK-34884 URL: https://issues.apache.org/jira/browse/SPARK-34884 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34542) Upgrade Parquet to 1.12.0

2021-03-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34542: Description: Parquet-1.12.0 release notes:

[jira] [Commented] (SPARK-34867) cannot resolve method when of Column() spark java

2021-03-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308644#comment-17308644 ] Yuming Wang commented on SPARK-34867: - How to reproduce this issue? > cannot resolve method when of

[jira] [Commented] (SPARK-34859) Vectorized parquet reader needs synchronization among pages for column index

2021-03-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308435#comment-17308435 ] Yuming Wang commented on SPARK-34859: - [~lxian2] Thank you for reporting this issue. Would you like

[jira] [Resolved] (SPARK-34778) Upgrade to Avro 1.10.2

2021-03-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34778. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31866

[jira] [Updated] (SPARK-34802) Move simplify expression rules before push down rules

2021-03-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34802: Summary: Move simplify expression rules before push down rules (was: Optimize Optimizer

[jira] [Issue Comment Deleted] (SPARK-34775) Push down limit through window when partitionSpec is not empty

2021-03-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34775: Comment: was deleted (was: User 'viirya' has created a pull request for this issue:

[jira] [Created] (SPARK-34814) LikeSimplification should handle NULL

2021-03-21 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34814: --- Summary: LikeSimplification should handle NULL Key: SPARK-34814 URL: https://issues.apache.org/jira/browse/SPARK-34814 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-34784) Upgrade Jackson to 2.12.2

2021-03-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34784: --- Assignee: Xingchao, Zhang > Upgrade Jackson to 2.12.2 > - > >

[jira] [Resolved] (SPARK-34784) Upgrade Jackson to 2.12.2

2021-03-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34784. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31878

[jira] [Resolved] (SPARK-34810) Update PostgreSQL test with the latest results

2021-03-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34810. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31910

[jira] [Assigned] (SPARK-34810) Update PostgreSQL test with the latest results

2021-03-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34810: --- Assignee: William Hyun > Update PostgreSQL test with the latest results >

[jira] [Assigned] (SPARK-28220) join foldable condition not pushed down when parent filter is totally pushed down

2021-03-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-28220: --- Assignee: Yuming Wang > join foldable condition not pushed down when parent filter is

[jira] [Resolved] (SPARK-28220) join foldable condition not pushed down when parent filter is totally pushed down

2021-03-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-28220. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31857

[jira] [Assigned] (SPARK-34808) Removes outer join if it only has distinct on streamed side

2021-03-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34808: --- Assignee: (was: Yuming Wang) > Removes outer join if it only has distinct on streamed

[jira] [Created] (SPARK-34808) Removes outer join if it only has distinct on streamed side

2021-03-20 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34808: --- Summary: Removes outer join if it only has distinct on streamed side Key: SPARK-34808 URL: https://issues.apache.org/jira/browse/SPARK-34808 Project: Spark

[jira] [Created] (SPARK-34807) Push down filter through window after TransposeWindow

2021-03-20 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34807: --- Summary: Push down filter through window after TransposeWindow Key: SPARK-34807 URL: https://issues.apache.org/jira/browse/SPARK-34807 Project: Spark Issue

[jira] [Created] (SPARK-34802) Optimize Optimizer defaultBatches rule order

2021-03-19 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34802: --- Summary: Optimize Optimizer defaultBatches rule order Key: SPARK-34802 URL: https://issues.apache.org/jira/browse/SPARK-34802 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-34798) Fix incorrect join condition

2021-03-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34798. - Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed

[jira] [Assigned] (SPARK-34798) Fix incorrect join condition

2021-03-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34798: --- Assignee: Hongyi Zhang > Fix incorrect join condition > > >

[jira] [Resolved] (SPARK-34772) RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

2021-03-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34772. - Fix Version/s: 3.0.3 3.1.2 3.2.0 Assignee:

[jira] [Updated] (SPARK-34798) Fix incorrect join condition

2021-03-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34798: Summary: Fix incorrect join condition (was: Replace `==` with `===` in join condition) > Fix

[jira] [Created] (SPARK-34798) Replace `==` with `===` in join condition

2021-03-18 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34798: --- Summary: Replace `==` with `===` in join condition Key: SPARK-34798 URL: https://issues.apache.org/jira/browse/SPARK-34798 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-34784) Upgrade Jackson to 2.12.2

2021-03-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34784: Parent: (was: SPARK-26345) Issue Type: Improvement (was: Sub-task) > Upgrade Jackson

[jira] [Updated] (SPARK-34728) Remove all SQLConf.get if extends from SQLConfHelper

2021-03-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34728: Description: We do not need SQLConf.get if extends from SQLConfHelper. For example:

[jira] [Resolved] (SPARK-34728) Remove all SQLConf.get if extends from SQLConfHelper

2021-03-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34728. - Fix Version/s: 3.2.0 Assignee: Xuedong Luan Resolution: Fixed Issue resolved by

[jira] [Updated] (SPARK-34728) Remove all SQLConf.get if extends from SQLConfHelper

2021-03-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34728: Summary: Remove all SQLConf.get if extends from SQLConfHelper (was: Remove all SQLConf.get if

[jira] [Updated] (SPARK-34784) Upgrade Jackson to 2.12.2

2021-03-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34784: Fix Version/s: (was: 3.2.0) > Upgrade Jackson to 2.12.2 > - > >

[jira] [Assigned] (SPARK-34784) Upgrade Jackson to 2.12.2

2021-03-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34784: --- Assignee: (was: Yuming Wang) > Upgrade Jackson to 2.12.2 > - >

[jira] [Created] (SPARK-34775) Push down limit through window when partitionSpec is not empty

2021-03-17 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34775: --- Summary: Push down limit through window when partitionSpec is not empty Key: SPARK-34775 URL: https://issues.apache.org/jira/browse/SPARK-34775 Project: Spark

[jira] [Reopened] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-21449: - > Hive client's SessionState was not closed properly in HiveExternalCatalog >

[jira] [Updated] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21449: Labels: (was: bulk-closed) > Hive client's SessionState was not closed properly in

[jira] [Resolved] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-21449. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31833

[jira] [Updated] (SPARK-23745) Remove the directories of the “hive.downloaded.resources.dir” when HiveThriftServer2 stopped

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-23745: Labels: (was: bulk-closed) > Remove the directories of the “hive.downloaded.resources.dir” when

[jira] [Resolved] (SPARK-23745) Remove the directories of the “hive.downloaded.resources.dir” when HiveThriftServer2 stopped

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-23745. - Fix Version/s: 3.2.0 Assignee: Kent Yao Resolution: Fixed Issue resolved by

[jira] [Reopened] (SPARK-23745) Remove the directories of the “hive.downloaded.resources.dir” when HiveThriftServer2 stopped

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-23745: - > Remove the directories of the “hive.downloaded.resources.dir” when > HiveThriftServer2 stopped >

[jira] [Assigned] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-21449: --- Assignee: Kent Yao > Hive client's SessionState was not closed properly in

[jira] [Updated] (SPARK-34744) Improve error message for casting cause overflow error

2021-03-15 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34744: Description: For example: {code:sql} set spark.sql.ansi.enabled=true; select tinyint(128) *

[jira] [Created] (SPARK-34744) Improve error message for casting cause overflow error

2021-03-15 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34744: --- Summary: Improve error message for casting cause overflow error Key: SPARK-34744 URL: https://issues.apache.org/jira/browse/SPARK-34744 Project: Spark Issue

[jira] [Issue Comment Deleted] (SPARK-34728) Remove all SQLConf.get if extends from Rule

2021-03-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34728: Comment: was deleted (was: I'm working on.) > Remove all SQLConf.get if extends from Rule >

[jira] [Resolved] (SPARK-34730) how use the idea and scala to develop the spark,(configuration environment)

2021-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34730. - Resolution: Invalid Please see IDE Setup:

[jira] [Updated] (SPARK-34730) how use the idea and scala to develop the spark,(configuration environment)

2021-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34730: Fix Version/s: (was: 3.0.1) > how use the idea and scala to develop the spark,(configuration

[jira] [Updated] (SPARK-34730) how use the idea and scala to develop the spark,(configuration environment)

2021-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34730: Target Version/s: (was: 3.0.1) > how use the idea and scala to develop the spark,(configuration

[jira] [Commented] (SPARK-34728) Remove all SQLConf.get if extends from Rule

2021-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300533#comment-17300533 ] Yuming Wang commented on SPARK-34728: - I'm working on. > Remove all SQLConf.get if extends from

[jira] [Created] (SPARK-34728) Remove all SQLConf.get if extends from Rule

2021-03-12 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34728: --- Summary: Remove all SQLConf.get if extends from Rule Key: SPARK-34728 URL: https://issues.apache.org/jira/browse/SPARK-34728 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-34444) Pushdown scalar-subquery filter to FileSourceScan

2021-03-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-3. - Resolution: Won't Fix > Pushdown scalar-subquery filter to FileSourceScan >

[jira] [Resolved] (SPARK-30303) Percentile will hit match err when percentage is null

2021-03-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-30303. - Fix Version/s: 3.0.0 Resolution: Fixed > Percentile will hit match err when percentage

[jira] [Updated] (SPARK-34631) Caught Hive MetaException when query by partition (partition col start with ‘$’)

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34631: Fix Version/s: (was: 2.4.4) > Caught Hive MetaException when query by partition (partition

[jira] [Updated] (SPARK-34631) Caught Hive MetaException when query by partition (partition col start with ‘$’)

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34631: Target Version/s: (was: 2.4.4) > Caught Hive MetaException when query by partition (partition

[jira] [Created] (SPARK-34628) Remove GlobalLimit operator if it's child max row <= limit

2021-03-04 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34628: --- Summary: Remove GlobalLimit operator if it's child max row <= limit Key: SPARK-34628 URL: https://issues.apache.org/jira/browse/SPARK-34628 Project: Spark

[jira] [Updated] (SPARK-34622) Push down limit through Project

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34622: Issue Type: Improvement (was: Bug) > Push down limit through Project >

[jira] [Updated] (SPARK-34622) Push down limit through Project

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34622: Summary: Push down limit through Project (was: Fix push down limit through join) > Push down

[jira] [Updated] (SPARK-34622) Fix push down limit through join

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34622: Description: We will insert a Project if the Join output is not match the LocalLimit. We can not

[jira] [Updated] (SPARK-34622) Fix push down limit through join

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34622: Description: We will insert a Project if the Join output is not match the LocalLimit.

[jira] [Updated] (SPARK-34622) Fix push down limit through join

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34622: Description: We will insert a Project if the Join output is not match the LocalLimit.

[jira] [Updated] (SPARK-34622) Fix push down limit through join

2021-03-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34622: Description: {code:scala} spark.sql("create table t1(a int, b int, c int) using parquet")

[jira] [Created] (SPARK-34622) Fix push down limit through join

2021-03-04 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34622: --- Summary: Fix push down limit through join Key: SPARK-34622 URL: https://issues.apache.org/jira/browse/SPARK-34622 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-34581) BoundAttribute issue after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Parent: (was: SPARK-33910) Issue Type: Bug (was: Sub-task) > BoundAttribute issue

[jira] [Commented] (SPARK-34581) BoundAttribute issue will occur after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293665#comment-17293665 ] Yuming Wang commented on SPARK-34581: - cc [~Ngone51] > BoundAttribute issue will occur after

[jira] [Updated] (SPARK-34581) BoundAttribute issue after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Summary: BoundAttribute issue after optimization by BooleanSimplification and

[jira] [Updated] (SPARK-34581) BoundAttribute issue will occur after optimizated by BooleanSimplification and PushFoldableIntoBranches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Description: BoundAttribute issue will occur after optimization by BooleanSimplification and

[jira] [Updated] (SPARK-34581) BoundAttribute issue will occur after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Summary: BoundAttribute issue will occur after optimization by BooleanSimplification and

[jira] [Updated] (SPARK-34581) BoundAttribute issue will occur after optimizated by BooleanSimplification and PushFoldableIntoBranches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Summary: BoundAttribute issue will occur after optimizated by BooleanSimplification and

[jira] [Updated] (SPARK-34581) BoundAttribute issue when pushing foldable into branches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Description: BoundAttribute issue after optimized by BooleanSimplification and 

[jira] [Updated] (SPARK-34581) BoundAttribute issue when pushing foldable into branches

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Description: How to reproduce this issue: {code:scala} spark.sql("CREATE TABLE t1 (a INT, b INT)

[jira] [Updated] (SPARK-34595) DPP support RLIKE

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34595: Fix Version/s: (was: 3.1.2) (was: 3.2.0) > DPP support RLIKE >

[jira] [Updated] (SPARK-34595) DPP support RLIKE

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34595: Fix Version/s: (was: 3.1.2) (was: 3.2.0) > DPP support RLIKE >

[jira] [Assigned] (SPARK-34595) DPP support RLIKE

2021-03-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34595: --- Assignee: (was: Yuming Wang) > DPP support RLIKE > - > >

[jira] [Updated] (SPARK-34581) BoundAttribute issue when pushing foldable into branches

2021-03-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34581: Description: How to reproduce this issue: {code:scala} spark.sql("CREATE TABLE t1 (a INT, b INT)

[jira] [Created] (SPARK-34581) BoundAttribute issue when pushing foldable into branches

2021-03-01 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34581: --- Summary: BoundAttribute issue when pushing foldable into branches Key: SPARK-34581 URL: https://issues.apache.org/jira/browse/SPARK-34581 Project: Spark Issue

[jira] [Created] (SPARK-34575) Push down limit through window when partitionSpec is empty

2021-02-28 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34575: --- Summary: Push down limit through window when partitionSpec is empty Key: SPARK-34575 URL: https://issues.apache.org/jira/browse/SPARK-34575 Project: Spark

[jira] [Commented] (SPARK-34562) Leverage parquet bloom filters

2021-02-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292256#comment-17292256 ] Yuming Wang commented on SPARK-34562: - [~h-vetinari] This is an example to build bloom filter:

[jira] [Updated] (SPARK-34562) Leverage parquet bloom filters

2021-02-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34562: Component/s: (was: Input/Output) SQL > Leverage parquet bloom filters >

[jira] [Comment Edited] (SPARK-34542) Upgrade Parquet to 1.12.0

2021-02-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292041#comment-17292041 ] Yuming Wang edited comment on SPARK-34542 at 2/27/21, 7:14 AM: --- Benchmark

[jira] [Commented] (SPARK-34542) Upgrade Parquet to 1.12.0

2021-02-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292041#comment-17292041 ] Yuming Wang commented on SPARK-34542: - {code:scala} /* * Licensed to the Apache Software Foundation

[jira] [Updated] (SPARK-34436) DPP support LIKE ANY/ALL

2021-02-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34436: Issue Type: Bug (was: Improvement) > DPP support LIKE ANY/ALL > > >

[jira] [Created] (SPARK-34542) Upgrade Parquet to 1.12.0

2021-02-25 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34542: --- Summary: Upgrade Parquet to 1.12.0 Key: SPARK-34542 URL: https://issues.apache.org/jira/browse/SPARK-34542 Project: Spark Issue Type: Improvement

[jira] [Issue Comment Deleted] (SPARK-33537) Hive Metastore filter pushdown improvement

2021-02-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33537: Comment: was deleted (was: User 'ulysses-you' has created a pull request for this issue:

[jira] [Resolved] (SPARK-26138) Pushdown limit through InnerLike when condition is empty

2021-02-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-26138. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31567

[jira] [Assigned] (SPARK-26138) Pushdown limit through InnerLike when condition is empty

2021-02-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-26138: --- Assignee: Yuming Wang > Pushdown limit through InnerLike when condition is empty >

[jira] [Updated] (SPARK-34512) Disable validate default values when parsing Avro schemas

2021-02-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34512: Description: How to reproduce this issue: {code:scala} // Add this test to

[jira] [Created] (SPARK-34512) Disable validate default values when parsing Avro schemas

2021-02-23 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34512: --- Summary: Disable validate default values when parsing Avro schemas Key: SPARK-34512 URL: https://issues.apache.org/jira/browse/SPARK-34512 Project: Spark

[jira] [Resolved] (SPARK-27500) Add tests for built-in Hive 2.3

2021-02-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-27500. - Fix Version/s: 3.0.0 Resolution: Fixed > Add tests for built-in Hive 2.3 >

[jira] [Assigned] (SPARK-27500) Add tests for built-in Hive 2.3

2021-02-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-27500: --- Assignee: Yuming Wang > Add tests for built-in Hive 2.3 > ---

[jira] [Resolved] (SPARK-33420) BroadCastJoin failure when keys on join side has cast from DateTyte to String

2021-02-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-33420. - Resolution: Duplicate > BroadCastJoin failure when keys on join side has cast from DateTyte to

[jira] [Commented] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-20 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287656#comment-17287656 ] Yuming Wang commented on SPARK-34479: - cc [~dongjoon] > Add zstandard codec to

[jira] [Comment Edited] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287463#comment-17287463 ] Yuming Wang edited comment on SPARK-34479 at 2/20/21, 3:02 AM: --- But zstd

[jira] [Comment Edited] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287463#comment-17287463 ] Yuming Wang edited comment on SPARK-34479 at 2/20/21, 3:02 AM: --- But zstd

[jira] [Commented] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287463#comment-17287463 ] Yuming Wang commented on SPARK-34479: - But zstd 1.4.5-12 is not compatible with 1.4.8-4.

[jira] [Updated] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34479: Description: Avro add zstandard codec since AVRO-2195. (was: Avro add AVRO-2195) > Add

[jira] [Updated] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34479: Description: Avro add AVRO-2195 > Add zstandard codec to spark.sql.avro.compression.codec >

[jira] [Created] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-02-19 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34479: --- Summary: Add zstandard codec to spark.sql.avro.compression.codec Key: SPARK-34479 URL: https://issues.apache.org/jira/browse/SPARK-34479 Project: Spark Issue

[jira] [Updated] (SPARK-34444) Pushdown scalar-subquery filter to FileSourceScan

2021-02-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-3: Description: We can pushdown {{a < (select max(d) from t2)}} to FileSourceScan: {code:scala}

<    6   7   8   9   10   11   12   13   14   15   >