[jira] [Resolved] (SPARK-35908) Remove repartition if the child maximum number of rows less than or equal to 1

2021-07-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35908. - Resolution: Not A Problem > Remove repartition if the child maximum number of rows less than or

[jira] [Updated] (SPARK-35967) Update nullability based on column statistics

2021-07-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35967: Summary: Update nullability based on column statistics (was: Update nullability base on column

[jira] [Created] (SPARK-35967) Update nullability base on column statistics

2021-07-01 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35967: --- Summary: Update nullability base on column statistics Key: SPARK-35967 URL: https://issues.apache.org/jira/browse/SPARK-35967 Project: Spark Issue Type:

[jira] [Updated] (SPARK-35904) Collapse above RebalancePartitions

2021-06-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35904: Fix Version/s: (was: 3.2.0) > Collapse above RebalancePartitions >

[jira] [Reopened] (SPARK-35904) Collapse above RebalancePartitions

2021-06-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-35904: - Reverted at https://github.com/apache/spark/commit/108635af1708173a72bec0e36bf3f2cea5b088c4 >

[jira] [Updated] (SPARK-35908) Remove repartition if the child maximum number of rows less than or equal to 1

2021-06-26 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35908: Description: {code:scala} spark.sql("select count(*) from range(1, 10, 2,

[jira] [Created] (SPARK-35908) Remove repartition if the child maximum number of rows less than or equal to 1

2021-06-26 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35908: --- Summary: Remove repartition if the child maximum number of rows less than or equal to 1 Key: SPARK-35908 URL: https://issues.apache.org/jira/browse/SPARK-35908

[jira] [Created] (SPARK-35906) Remove order by if the maximum number of rows less than or equal to 1

2021-06-26 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35906: --- Summary: Remove order by if the maximum number of rows less than or equal to 1 Key: SPARK-35906 URL: https://issues.apache.org/jira/browse/SPARK-35906 Project: Spark

[jira] [Updated] (SPARK-35904) Collapse above RebalancePartitions

2021-06-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35904: Description: Make RebalancePartitions extends RepartitionOperation. > Collapse above

[jira] [Created] (SPARK-35904) Collapse above RebalancePartitions

2021-06-25 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35904: --- Summary: Collapse above RebalancePartitions Key: SPARK-35904 URL: https://issues.apache.org/jira/browse/SPARK-35904 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-35886) Codegen issue for decimal type

2021-06-24 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35886: --- Summary: Codegen issue for decimal type Key: SPARK-35886 URL: https://issues.apache.org/jira/browse/SPARK-35886 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-34807) Push down filter through window after TransposeWindow

2021-06-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34807: --- Assignee: Tanel Kiis > Push down filter through window after TransposeWindow >

[jira] [Resolved] (SPARK-34807) Push down filter through window after TransposeWindow

2021-06-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34807. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31980

[jira] [Updated] (SPARK-35837) Recommendations for Common Query Problems

2021-06-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35837: Description: Teradata supportsĀ [Recommendations for Common Query

[jira] [Updated] (SPARK-35837) Recommendations for Common Query Problems

2021-06-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35837: Description: Teradata supportsĀ [Recommendations for Common Query

[jira] [Updated] (SPARK-35837) Recommendations for Common Query Problems

2021-06-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35837: Description: Teradata supportsĀ [Recommendations for Common Query

[jira] [Created] (SPARK-35837) Recommendations for Common Query Problems

2021-06-21 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35837: --- Summary: Recommendations for Common Query Problems Key: SPARK-35837 URL: https://issues.apache.org/jira/browse/SPARK-35837 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-35797) patternToRegex failed when pattern is star

2021-06-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17366080#comment-17366080 ] Yuming Wang commented on SPARK-35797: - How to reproduce this issue? > patternToRegex failed when

[jira] [Assigned] (SPARK-34120) Improve the statistics estimation

2021-06-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34120: --- Assignee: Yuming Wang > Improve the statistics estimation >

[jira] [Resolved] (SPARK-34120) Improve the statistics estimation

2021-06-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34120. - Fix Version/s: 3.2.0 Resolution: Fixed > Improve the statistics estimation >

[jira] [Assigned] (SPARK-35185) Improve Distinct statistics estimation

2021-06-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35185: --- Assignee: Yuming Wang > Improve Distinct statistics estimation >

[jira] [Resolved] (SPARK-35185) Improve Distinct statistics estimation

2021-06-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35185. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32291

[jira] [Updated] (SPARK-35786) Support optimize repartition by expression in AQE

2021-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35786: Parent: SPARK-35793 Issue Type: Sub-task (was: Improvement) > Support optimize

[jira] [Updated] (SPARK-30538) A not very elegant way to control ouput small file

2021-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-30538: Parent: SPARK-35793 Issue Type: Sub-task (was: Improvement) > A not very elegant way to

[jira] [Updated] (SPARK-35335) Improve CoalesceShufflePartitions to avoid generating small files

2021-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35335: Parent: SPARK-35793 Issue Type: Sub-task (was: Improvement) > Improve

[jira] [Updated] (SPARK-35650) Coalesce small output files through AQE

2021-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35650: Parent: SPARK-35793 Issue Type: Sub-task (was: New Feature) > Coalesce small output

[jira] [Updated] (SPARK-35725) Support repartition expand partitions in AQE

2021-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35725: Parent Issue: SPARK-35793 (was: SPARK-33828) > Support repartition expand partitions in AQE >

[jira] [Updated] (SPARK-31264) Repartition by dynamic partition columns before insert table

2021-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31264: Parent: SPARK-35793 Issue Type: Sub-task (was: Improvement) > Repartition by dynamic

[jira] [Created] (SPARK-35793) Repartition before writing data source tables

2021-06-17 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35793: --- Summary: Repartition before writing data source tables Key: SPARK-35793 URL: https://issues.apache.org/jira/browse/SPARK-35793 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-35556) Remove the close HiveClient's SessionState

2021-06-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35556. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32693

[jira] [Assigned] (SPARK-35556) Remove the close HiveClient's SessionState

2021-06-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35556: --- Assignee: Yang Jie > Remove the close HiveClient's SessionState >

[jira] [Updated] (SPARK-35556) Avoid log NoSuchMethodError when HiveClientImpl.state close

2021-06-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35556: Component/s: (was: Tests) > Avoid log NoSuchMethodError when HiveClientImpl.state close >

[jira] [Updated] (SPARK-35556) Remove the close HiveClient's SessionState

2021-06-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35556: Summary: Remove the close HiveClient's SessionState (was: Avoid log NoSuchMethodError when

[jira] [Updated] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2021-06-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28560: Attachment: localShuffleReader.png > Optimize shuffle reader to local shuffle reader when smj

[jira] [Commented] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2021-06-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364079#comment-17364079 ] Yuming Wang commented on SPARK-28560: - This is very useful if there is data skew at the probe side.

[jira] [Updated] (SPARK-27714) Support Join Reorder based on Genetic Algorithm when the # of joined tables > 12

2021-06-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27714: Target Version/s: (was: 3.0.0) > Support Join Reorder based on Genetic Algorithm when the # of

[jira] [Assigned] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-06-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35321: --- Assignee: Chao Sun > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions

[jira] [Resolved] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-06-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35321. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32887

[jira] [Created] (SPARK-35650) Coalesce small output files through AQE

2021-06-04 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35650: --- Summary: Coalesce small output files through AQE Key: SPARK-35650 URL: https://issues.apache.org/jira/browse/SPARK-35650 Project: Spark Issue Type: New

[jira] [Assigned] (SPARK-34808) Removes outer join if it only has distinct on streamed side

2021-05-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34808: --- Assignee: Yuming Wang > Removes outer join if it only has distinct on streamed side >

[jira] [Resolved] (SPARK-34808) Removes outer join if it only has distinct on streamed side

2021-05-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34808. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31908

[jira] [Updated] (SPARK-35571) tag v3.0.0 org.apache.spark.sql.catalyst.parser.AstBuilder import error

2021-05-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35571: Target Version/s: (was: 3.0.0) > tag v3.0.0 org.apache.spark.sql.catalyst.parser.AstBuilder

[jira] [Commented] (SPARK-35568) UnsupportedOperationException: WholeStageCodegen (3) does not implement doExecuteBroadcast

2021-05-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354174#comment-17354174 ] Yuming Wang commented on SPARK-35568: - Thank you [~dongjoon]. > UnsupportedOperationException:

[jira] [Updated] (SPARK-35568) UnsupportedOperationException: WholeStageCodegen (3) does not implement doExecuteBroadcast

2021-05-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35568: Description: How to reproduce: {code:scala} sql( """ |SELECT s.store_id,

[jira] [Created] (SPARK-35568) UnsupportedOperationException: WholeStageCodegen (3) does not implement doExecuteBroadcast

2021-05-30 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35568: --- Summary: UnsupportedOperationException: WholeStageCodegen (3) does not implement doExecuteBroadcast Key: SPARK-35568 URL: https://issues.apache.org/jira/browse/SPARK-35568

[jira] [Commented] (SPARK-35441) InMemoryFileIndex load all files into memroy

2021-05-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350242#comment-17350242 ] Yuming Wang commented on SPARK-35441: - Please increase your driver memory. or you should merge small

[jira] [Updated] (SPARK-35494) Timestamp casting performance issue when invoked with timezone

2021-05-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35494: Target Version/s: (was: 2.4.9) > Timestamp casting performance issue when invoked with timezone

[jira] [Commented] (SPARK-32291) COALESCE should not reduce the child parallelism if it is Join

2021-05-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349930#comment-17349930 ] Yuming Wang commented on SPARK-32291: - We can use localCheckpoint to workaround this issue:

[jira] [Assigned] (SPARK-35244) invoke should throw the original exception

2021-05-21 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35244: --- Assignee: Wenchen Fan (was: Apache Spark) > invoke should throw the original exception >

[jira] [Commented] (SPARK-35441) InMemoryFileIndex load all files into memroy

2021-05-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347404#comment-17347404 ] Yuming Wang commented on SPARK-35441: - What is your driver memory? > InMemoryFileIndex load all

[jira] [Created] (SPARK-35415) Change information to map type for SHOW TABLE EXTENDED command

2021-05-16 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35415: --- Summary: Change information to map type for SHOW TABLE EXTENDED command Key: SPARK-35415 URL: https://issues.apache.org/jira/browse/SPARK-35415 Project: Spark

[jira] [Resolved] (SPARK-35286) Replace SessionState.start with SessionState.setCurrentSessionState

2021-05-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35286. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32410

[jira] [Assigned] (SPARK-35286) Replace SessionState.start with SessionState.setCurrentSessionState

2021-05-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35286: --- Assignee: Yuming Wang > Replace SessionState.start with

[jira] [Commented] (SPARK-35365) spark3.1.1 use too long time to analyze table fields

2021-05-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342981#comment-17342981 ] Yuming Wang commented on SPARK-35365: - {noformat} -- 2.4

[jira] [Commented] (SPARK-35365) spark3.1.1 use too long to analyze table fields

2021-05-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342354#comment-17342354 ] Yuming Wang commented on SPARK-35365: - [~xiaohua] Could you check which rule affect the performance,

[jira] [Created] (SPARK-35335) Improve CoalesceShufflePartitions to avoid generating small files

2021-05-07 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35335: --- Summary: Improve CoalesceShufflePartitions to avoid generating small files Key: SPARK-35335 URL: https://issues.apache.org/jira/browse/SPARK-35335 Project: Spark

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-05-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet")

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339944#comment-17339944 ] Yuming Wang commented on SPARK-35321: - Could we add a parameter to disable registerAllFunctionsOnce?

[jira] [Resolved] (SPARK-35315) Keep benchmark result consistent between spark-submit and SBT

2021-05-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35315. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32440

[jira] [Assigned] (SPARK-35315) Keep benchmark result consistent between spark-submit and SBT

2021-05-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35315: --- Assignee: Chao Sun > Keep benchmark result consistent between spark-submit and SBT >

[jira] [Updated] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-05-04 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35316: Description: It will not pushdown filters for In/InSet predicates: {code:scala}

[jira] [Created] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-05-04 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35316: --- Summary: UnwrapCastInBinaryComparison support In/InSet predicate Key: SPARK-35316 URL: https://issues.apache.org/jira/browse/SPARK-35316 Project: Spark Issue

[jira] [Created] (SPARK-35286) Replace SessionState.start with SessionState.setCurrentSessionState

2021-05-01 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35286: --- Summary: Replace SessionState.start with SessionState.setCurrentSessionState Key: SPARK-35286 URL: https://issues.apache.org/jira/browse/SPARK-35286 Project: Spark

[jira] [Commented] (SPARK-35245) DynamicFilter pushdown not working

2021-05-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337807#comment-17337807 ] Yuming Wang commented on SPARK-35245: - This is because filtering side do not has selective predicate

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet")

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet")

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet")

[jira] [Created] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35273: --- Summary: CombineFilters support non-deterministic expressions Key: SPARK-35273 URL: https://issues.apache.org/jira/browse/SPARK-35273 Project: Spark Issue

[jira] [Created] (SPARK-35251) Improve LiveEntityHelpers.newAccumulatorInfos performace

2021-04-27 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35251: --- Summary: Improve LiveEntityHelpers.newAccumulatorInfos performace Key: SPARK-35251 URL: https://issues.apache.org/jira/browse/SPARK-35251 Project: Spark Issue

[jira] [Commented] (SPARK-34897) Support reconcile schemas based on index after nested column pruning

2021-04-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331225#comment-17331225 ] Yuming Wang commented on SPARK-34897: - Issue resolved by pull request 31993

[jira] [Resolved] (SPARK-34897) Support reconcile schemas based on index after nested column pruning

2021-04-24 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34897. - Fix Version/s: 3.2.0 3.1.2 3.0.3 Assignee: Yuming

[jira] [Created] (SPARK-35203) Improve Repartition statistics estimation

2021-04-23 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35203: --- Summary: Improve Repartition statistics estimation Key: SPARK-35203 URL: https://issues.apache.org/jira/browse/SPARK-35203 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-35191) all columns are read even if column pruning applies when spark3.0 read table written by spark2.2

2021-04-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329932#comment-17329932 ] Yuming Wang commented on SPARK-35191: - Could you check if it works after

[jira] [Created] (SPARK-35185) Improve Distinct statistics estimation

2021-04-22 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35185: --- Summary: Improve Distinct statistics estimation Key: SPARK-35185 URL: https://issues.apache.org/jira/browse/SPARK-35185 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-35121) Improve JoinSelection when join condition is not defined

2021-04-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35121: Summary: Improve JoinSelection when join condition is not defined (was: Improve JoinSelection

[jira] [Created] (SPARK-35121) Improve JoinSelection when join condition is empty

2021-04-17 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35121: --- Summary: Improve JoinSelection when join condition is empty Key: SPARK-35121 URL: https://issues.apache.org/jira/browse/SPARK-35121 Project: Spark Issue Type:

[jira] [Created] (SPARK-35118) Propagate empty relation through Join if join condition is empty

2021-04-17 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35118: --- Summary: Propagate empty relation through Join if join condition is empty Key: SPARK-35118 URL: https://issues.apache.org/jira/browse/SPARK-35118 Project: Spark

[jira] [Updated] (SPARK-34087) a memory leak occurs when we clone the spark session

2021-04-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34087: Attachment: screenshot-1.png > a memory leak occurs when we clone the spark session >

[jira] [Updated] (SPARK-34087) a memory leak occurs when we clone the spark session

2021-04-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34087: Attachment: (was: screenshot-1.png) > a memory leak occurs when we clone the spark session >

[jira] [Updated] (SPARK-34897) Support reconcile schemas based on index after nested column pruning

2021-04-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34897: Summary: Support reconcile schemas based on index after nested column pruning (was: The given

[jira] [Issue Comment Deleted] (SPARK-34897) Support reconcile schemas based on index after nested column pruning

2021-04-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34897: Comment: was deleted (was: We can workaround this issue by setting

[jira] [Resolved] (SPARK-35007) Spark 2.4.x version does not support numeric

2021-04-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35007. - Resolution: Won't Fix > Spark 2.4.x version does not support numeric >

[jira] [Updated] (SPARK-35007) Spark 2.4.x version does not support numeric

2021-04-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35007: Target Version/s: (was: 2.4.8) > Spark 2.4.x version does not support numeric >

[jira] [Commented] (SPARK-35010) nestedSchemaPruning causes issue when reading hive generated Orc files

2021-04-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17318347#comment-17318347 ] Yuming Wang commented on SPARK-35010: - Yes. It is an issue:

[jira] [Updated] (SPARK-35002) Fix the java.net.BindException when testing with Github Action

2021-04-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35002: Summary: Fix the java.net.BindException when testing with Github Action (was: Try to fix the

[jira] [Created] (SPARK-35002) Try to fix the java.net.BindException when testing with Github Action

2021-04-08 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35002: --- Summary: Try to fix the java.net.BindException when testing with Github Action Key: SPARK-35002 URL: https://issues.apache.org/jira/browse/SPARK-35002 Project: Spark

[jira] [Resolved] (SPARK-34966) Avoid shuffle if join type do not match

2021-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34966. - Resolution: Invalid

[jira] [Commented] (SPARK-34967) Regression in spark 3.1.1 for window function and struct binding resolution

2021-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315564#comment-17315564 ] Yuming Wang commented on SPARK-34967: - How to reproduce this issue? > Regression in spark 3.1.1 for

[jira] [Created] (SPARK-34966) Avoid shuffle if join type do not match

2021-04-06 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34966: --- Summary: Avoid shuffle if join type do not match Key: SPARK-34966 URL: https://issues.apache.org/jira/browse/SPARK-34966 Project: Spark Issue Type:

[jira] [Updated] (SPARK-33979) Filter predicate reorder

2021-04-02 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33979: Description: Reorder filter predicate to improve query performance: {noformat} others < In < Like

[jira] [Issue Comment Deleted] (SPARK-34931) CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.

2021-04-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34931: Comment: was deleted (was: User 'sarutak' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-34931) CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.

2021-04-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34931: Comment: was deleted (was: User 'HyukjinKwon' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-34931) CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.

2021-04-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34931: Comment: was deleted (was: User 'sarutak' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-34931) CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.

2021-04-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34931: Comment: was deleted (was: User 'HyukjinKwon' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-34931) CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.

2021-04-01 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34931: Comment: was deleted (was: User 'sarutak' has created a pull request for this issue:

[jira] [Updated] (SPARK-34920) Introduce SQLSTATE and ERRORCODE to SQL Exception

2021-03-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34920: Description: SQLSTATE is SQL standard state. Please see was: SQLSTATE is SQL standard state

[jira] [Created] (SPARK-34920) Introduce SQLSTATE and ERRORCODE to SQL Exception

2021-03-31 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34920: --- Summary: Introduce SQLSTATE and ERRORCODE to SQL Exception Key: SPARK-34920 URL: https://issues.apache.org/jira/browse/SPARK-34920 Project: Spark Issue Type:

[jira] [Commented] (SPARK-34897) The given data schema has less fields than the actual ORC physical schema

2021-03-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310500#comment-17310500 ] Yuming Wang commented on SPARK-34897: - We can workaround this issue by setting

[jira] [Updated] (SPARK-34897) The given data schema has less fields than the actual ORC physical schema

2021-03-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34897: Description: How to reproduce this issue: {code:scala} spark.sql( """ |CREATE TABLE `t1` (

[jira] [Created] (SPARK-34897) The given data schema has less fields than the actual ORC physical schema

2021-03-29 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34897: --- Summary: The given data schema has less fields than the actual ORC physical schema Key: SPARK-34897 URL: https://issues.apache.org/jira/browse/SPARK-34897 Project:

<    5   6   7   8   9   10   11   12   13   14   >