[jira] [Created] (SPARK-36511) Remove ColumnIO once PARQUET-2050 is released in Parquet 1.13

2021-08-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-36511: Summary: Remove ColumnIO once PARQUET-2050 is released in Parquet 1.13 Key: SPARK-36511 URL: https://issues.apache.org/jira/browse/SPARK-36511 Project: Spark Issue

[jira] [Issue Comment Deleted] (SPARK-34861) Support nested column in Spark vectorized readers

2021-08-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34861: - Comment: was deleted (was: Synced with [~chengsu] offline and I will take over this JIRA.) > Support

[jira] [Commented] (SPARK-36440) Spark3 fails to read hive table with mixed format

2021-08-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394529#comment-17394529 ] Chao Sun commented on SPARK-36440: -- Hmm really? Spark 2.x support this? I'm not sure why Spark is still

[jira] [Commented] (SPARK-36317) PruneFileSourcePartitionsSuite tests are failing after the fix to SPARK-36136

2021-07-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388204#comment-17388204 ] Chao Sun commented on SPARK-36317: -- [~vsowrirajan]: the change is already reverted - are you still

[jira] [Updated] (SPARK-36137) HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

2021-07-14 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-36137: - Description: At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use

[jira] [Created] (SPARK-36137) HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

2021-07-14 Thread Chao Sun (Jira)
Chao Sun created SPARK-36137: Summary: HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS Key: SPARK-36137 URL: https://issues.apache.org/jira/browse/SPARK-36137

[jira] [Created] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

2021-07-14 Thread Chao Sun (Jira)
Chao Sun created SPARK-36136: Summary: Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive Key: SPARK-36136 URL: https://issues.apache.org/jira/browse/SPARK-36136 Project: Spark

[jira] [Commented] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

2021-07-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380326#comment-17380326 ] Chao Sun commented on SPARK-36128: -- Thanks, I'm slightly inclined to reuse the existing config but

[jira] [Created] (SPARK-36131) Refactor ParquetColumnIndexSuite

2021-07-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-36131: Summary: Refactor ParquetColumnIndexSuite Key: SPARK-36131 URL: https://issues.apache.org/jira/browse/SPARK-36131 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

2021-07-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380299#comment-17380299 ] Chao Sun commented on SPARK-36128: -- [~hyukjin.kwon] you are right - I didn't know this config is

[jira] [Comment Edited] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

2021-07-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380299#comment-17380299 ] Chao Sun edited comment on SPARK-36128 at 7/14/21, 4:24 AM: [~hyukjin.kwon]

[jira] [Created] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

2021-07-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-36128: Summary: CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning Key: SPARK-36128 URL: https://issues.apache.org/jira/browse/SPARK-36128

[jira] [Updated] (SPARK-36123) Parquet vectorized reader doesn't skip null values correctly

2021-07-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-36123: - Labels: correctness (was: ) > Parquet vectorized reader doesn't skip null values correctly >

[jira] [Created] (SPARK-36123) Parquet vectorized reader doesn't skip null values correctly

2021-07-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-36123: Summary: Parquet vectorized reader doesn't skip null values correctly Key: SPARK-36123 URL: https://issues.apache.org/jira/browse/SPARK-36123 Project: Spark Issue

[jira] [Updated] (SPARK-35743) Improve Parquet vectorized reader

2021-07-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35743: - Labels: parquet (was: ) > Improve Parquet vectorized reader > - > >

[jira] [Created] (SPARK-36056) Combine readBatch and readIntegers in VectorizedRleValuesReader

2021-07-08 Thread Chao Sun (Jira)
Chao Sun created SPARK-36056: Summary: Combine readBatch and readIntegers in VectorizedRleValuesReader Key: SPARK-36056 URL: https://issues.apache.org/jira/browse/SPARK-36056 Project: Spark

[jira] [Created] (SPARK-35959) Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

2021-06-30 Thread Chao Sun (Jira)
Chao Sun created SPARK-35959: Summary: Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions Key: SPARK-35959 URL: https://issues.apache.org/jira/browse/SPARK-35959 Project: Spark

[jira] [Created] (SPARK-35867) Enable vectorized read for VectorizedPlainValuesReader.readBooleans

2021-06-23 Thread Chao Sun (Jira)
Chao Sun created SPARK-35867: Summary: Enable vectorized read for VectorizedPlainValuesReader.readBooleans Key: SPARK-35867 URL: https://issues.apache.org/jira/browse/SPARK-35867 Project: Spark

[jira] [Updated] (SPARK-35846) Introduce ParquetReadState to track various states while reading a Parquet column chunk

2021-06-21 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35846: - Description: This is mostly refactoring work to complete SPARK-34859 > Introduce ParquetReadState to

[jira] [Created] (SPARK-35846) Introduce ParquetReadState to track various states while reading a Parquet column chunk

2021-06-21 Thread Chao Sun (Jira)
Chao Sun created SPARK-35846: Summary: Introduce ParquetReadState to track various states while reading a Parquet column chunk Key: SPARK-35846 URL: https://issues.apache.org/jira/browse/SPARK-35846

[jira] [Updated] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35640: - Parent: SPARK-35743 Issue Type: Sub-task (was: Improvement) > Refactor Parquet vectorized

[jira] [Created] (SPARK-35743) Improve Parquet vectorized reader

2021-06-11 Thread Chao Sun (Jira)
Chao Sun created SPARK-35743: Summary: Improve Parquet vectorized reader Key: SPARK-35743 URL: https://issues.apache.org/jira/browse/SPARK-35743 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-34861) Support nested column in Spark vectorized readers

2021-06-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361090#comment-17361090 ] Chao Sun commented on SPARK-34861: -- Synced with [~chengsu] offline and I will take over this JIRA. >

[jira] [Updated] (SPARK-35703) Remove HashClusteredDistribution

2021-06-09 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35703: - Description: Currently Spark has {{HashClusteredDistribution}} and {{ClusteredDistribution}}. The only

[jira] [Created] (SPARK-35703) Remove HashClusteredDistribution

2021-06-09 Thread Chao Sun (Jira)
Chao Sun created SPARK-35703: Summary: Remove HashClusteredDistribution Key: SPARK-35703 URL: https://issues.apache.org/jira/browse/SPARK-35703 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-03 Thread Chao Sun (Jira)
Chao Sun created SPARK-35640: Summary: Refactor Parquet vectorized reader to remove duplicated code paths Key: SPARK-35640 URL: https://issues.apache.org/jira/browse/SPARK-35640 Project: Spark

[jira] [Updated] (SPARK-34859) Vectorized parquet reader needs synchronization among pages for column index

2021-05-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34859: - Priority: Critical (was: Major) > Vectorized parquet reader needs synchronization among pages for

[jira] [Updated] (SPARK-34859) Vectorized parquet reader needs synchronization among pages for column index

2021-05-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34859: - Labels: correctness (was: ) > Vectorized parquet reader needs synchronization among pages for column

[jira] [Commented] (SPARK-35461) Error when reading dictionary-encoded Parquet int column when read schema is bigint

2021-05-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348667#comment-17348667 ] Chao Sun commented on SPARK-35461: -- Actually this also fails when turning off the vectorized reader:

[jira] [Created] (SPARK-35461) Error when reading dictionary-encoded Parquet int column when read schema is bigint

2021-05-20 Thread Chao Sun (Jira)
Chao Sun created SPARK-35461: Summary: Error when reading dictionary-encoded Parquet int column when read schema is bigint Key: SPARK-35461 URL: https://issues.apache.org/jira/browse/SPARK-35461 Project:

[jira] [Commented] (SPARK-35422) Many test cases failed in Scala 2.13 CI

2021-05-17 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346448#comment-17346448 ] Chao Sun commented on SPARK-35422: -- Thanks [~dongjoon]. I've opened a PR for the above failures:

[jira] [Created] (SPARK-35390) Handle type coercion when resolving V2 functions

2021-05-12 Thread Chao Sun (Jira)
Chao Sun created SPARK-35390: Summary: Handle type coercion when resolving V2 functions Key: SPARK-35390 URL: https://issues.apache.org/jira/browse/SPARK-35390 Project: Spark Issue Type:

[jira] [Created] (SPARK-35389) Analyzer should set progagateNull to false for magic function invocation

2021-05-12 Thread Chao Sun (Jira)
Chao Sun created SPARK-35389: Summary: Analyzer should set progagateNull to false for magic function invocation Key: SPARK-35389 URL: https://issues.apache.org/jira/browse/SPARK-35389 Project: Spark

[jira] [Updated] (SPARK-35384) Improve performance for InvokeLike.invoke

2021-05-12 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35384: - Issue Type: Improvement (was: Bug) > Improve performance for InvokeLike.invoke >

[jira] [Created] (SPARK-35384) Improve performance for InvokeLike.invoke

2021-05-12 Thread Chao Sun (Jira)
Chao Sun created SPARK-35384: Summary: Improve performance for InvokeLike.invoke Key: SPARK-35384 URL: https://issues.apache.org/jira/browse/SPARK-35384 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-35361) Improve performance for ApplyFunctionExpression

2021-05-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35361: - Priority: Minor (was: Major) > Improve performance for ApplyFunctionExpression >

[jira] [Updated] (SPARK-35361) Improve performance for ApplyFunctionExpression

2021-05-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35361: - Priority: Major (was: Minor) > Improve performance for ApplyFunctionExpression >

[jira] [Created] (SPARK-35361) Improve performance for ApplyFunctionExpression

2021-05-10 Thread Chao Sun (Jira)
Chao Sun created SPARK-35361: Summary: Improve performance for ApplyFunctionExpression Key: SPARK-35361 URL: https://issues.apache.org/jira/browse/SPARK-35361 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35321: - Issue Type: Bug (was: Improvement) > Spark 3.x can't talk to HMS 1.2.x and lower due to

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339946#comment-17339946 ] Chao Sun commented on SPARK-35321: -- [~yumwang] I'm thinking of using 

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339906#comment-17339906 ] Chao Sun commented on SPARK-35321: -- [~xkrogen] yes that can help to solve the issue, but users need to

[jira] [Created] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
Chao Sun created SPARK-35321: Summary: Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing Key: SPARK-35321 URL: https://issues.apache.org/jira/browse/SPARK-35321

[jira] [Updated] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35321: - Description: https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API

[jira] [Updated] (SPARK-35315) Keep benchmark result consistent between spark-submit and SBT

2021-05-04 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35315: - Priority: Minor (was: Major) > Keep benchmark result consistent between spark-submit and SBT >

[jira] [Created] (SPARK-35315) Keep benchmark result consistent between spark-submit and SBT

2021-05-04 Thread Chao Sun (Jira)
Chao Sun created SPARK-35315: Summary: Keep benchmark result consistent between spark-submit and SBT Key: SPARK-35315 URL: https://issues.apache.org/jira/browse/SPARK-35315 Project: Spark Issue

[jira] [Updated] (SPARK-35281) StaticInvoke should not apply boxing if return type is primitive

2021-04-30 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35281: - Priority: Minor (was: Major) > StaticInvoke should not apply boxing if return type is primitive >

[jira] [Created] (SPARK-35281) StaticInvoke should not apply boxing if return type is primitive

2021-04-30 Thread Chao Sun (Jira)
Chao Sun created SPARK-35281: Summary: StaticInvoke should not apply boxing if return type is primitive Key: SPARK-35281 URL: https://issues.apache.org/jira/browse/SPARK-35281 Project: Spark

[jira] [Created] (SPARK-35261) Support static invoke for stateless UDF

2021-04-28 Thread Chao Sun (Jira)
Chao Sun created SPARK-35261: Summary: Support static invoke for stateless UDF Key: SPARK-35261 URL: https://issues.apache.org/jira/browse/SPARK-35261 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-34981) Implement V2 function resolution and evaluation

2021-04-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34981: - Parent: SPARK-35260 Issue Type: Sub-task (was: Improvement) > Implement V2 function resolution

[jira] [Created] (SPARK-35260) DataSourceV2 Function Catalog implementation

2021-04-28 Thread Chao Sun (Jira)
Chao Sun created SPARK-35260: Summary: DataSourceV2 Function Catalog implementation Key: SPARK-35260 URL: https://issues.apache.org/jira/browse/SPARK-35260 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-35233) Switch from bintray to scala.jfrog.io for SBT download in branch 2.4 and 3.0

2021-04-26 Thread Chao Sun (Jira)
Chao Sun created SPARK-35233: Summary: Switch from bintray to scala.jfrog.io for SBT download in branch 2.4 and 3.0 Key: SPARK-35233 URL: https://issues.apache.org/jira/browse/SPARK-35233 Project: Spark

[jira] [Created] (SPARK-35232) Nested column pruning should retain column metadata

2021-04-26 Thread Chao Sun (Jira)
Chao Sun created SPARK-35232: Summary: Nested column pruning should retain column metadata Key: SPARK-35232 URL: https://issues.apache.org/jira/browse/SPARK-35232 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-35195) Move InMemoryTable etc to org.apache.spark.sql.connector.catalog

2021-04-22 Thread Chao Sun (Jira)
Chao Sun created SPARK-35195: Summary: Move InMemoryTable etc to org.apache.spark.sql.connector.catalog Key: SPARK-35195 URL: https://issues.apache.org/jira/browse/SPARK-35195 Project: Spark

[jira] [Created] (SPARK-35003) Improve performance for reading smallint in vectorized Parquet reader

2021-04-09 Thread Chao Sun (Jira)
Chao Sun created SPARK-35003: Summary: Improve performance for reading smallint in vectorized Parquet reader Key: SPARK-35003 URL: https://issues.apache.org/jira/browse/SPARK-35003 Project: Spark

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-04-07 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316714#comment-17316714 ] Chao Sun commented on SPARK-34780: -- Hi [~mikechen] (and sorry for the late reply again), thanks for

[jira] [Commented] (SPARK-34981) Implement V2 function resolution and evaluation

2021-04-07 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316477#comment-17316477 ] Chao Sun commented on SPARK-34981: -- Will submit a PR soon. > Implement V2 function resolution and

[jira] [Created] (SPARK-34981) Implement V2 function resolution and evaluation

2021-04-07 Thread Chao Sun (Jira)
Chao Sun created SPARK-34981: Summary: Implement V2 function resolution and evaluation Key: SPARK-34981 URL: https://issues.apache.org/jira/browse/SPARK-34981 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34973) Cleanup unused fields and methods in vectorized Parquet reader

2021-04-06 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34973: - Priority: Minor (was: Major) > Cleanup unused fields and methods in vectorized Parquet reader >

[jira] [Created] (SPARK-34973) Cleanup unused fields and methods in vectorized Parquet reader

2021-04-06 Thread Chao Sun (Jira)
Chao Sun created SPARK-34973: Summary: Cleanup unused fields and methods in vectorized Parquet reader Key: SPARK-34973 URL: https://issues.apache.org/jira/browse/SPARK-34973 Project: Spark

[jira] [Created] (SPARK-34947) Streaming write to a V2 table should invalidate its associated cache

2021-04-02 Thread Chao Sun (Jira)
Chao Sun created SPARK-34947: Summary: Streaming write to a V2 table should invalidate its associated cache Key: SPARK-34947 URL: https://issues.apache.org/jira/browse/SPARK-34947 Project: Spark

[jira] [Created] (SPARK-34945) Fix Javadoc for catalyst module

2021-04-02 Thread Chao Sun (Jira)
Chao Sun created SPARK-34945: Summary: Fix Javadoc for catalyst module Key: SPARK-34945 URL: https://issues.apache.org/jira/browse/SPARK-34945 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308854#comment-17308854 ] Chao Sun commented on SPARK-34780: -- [~mikechen], yes you're right. I'm not sure if this is a big

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308262#comment-17308262 ] Chao Sun commented on SPARK-34780: -- Sorry for the late reply [~mikechen]! There's something I still not

[jira] [Commented] (SPARK-30497) migrate DESCRIBE TABLE to the new framework

2021-03-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308067#comment-17308067 ] Chao Sun commented on SPARK-30497: -- [~cloud_fan] this is resolved right? > migrate DESCRIBE TABLE to

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-19 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305109#comment-17305109 ] Chao Sun commented on SPARK-34780: -- Thanks for the reporting [~mikechen], the test case you provided is

[jira] [Updated] (SPARK-32703) Replace deprecated API calls from SpecificParquetRecordReaderBase

2021-02-26 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32703: - Description: Currently in {{SpecificParquetRecordReaderBase}} we use deprecated APIs in a few places

[jira] [Updated] (SPARK-32703) Replace deprecated API calls from SpecificParquetRecordReaderBase

2021-02-26 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32703: - Summary: Replace deprecated API calls from SpecificParquetRecordReaderBase (was: Enable dictionary

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

2021-02-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290707#comment-17290707 ] Chao Sun commented on SPARK-33212: -- Yes. I think the only class Spark needs from this jar is

[jira] [Comment Edited] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

2021-02-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290613#comment-17290613 ] Chao Sun edited comment on SPARK-33212 at 2/25/21, 2:21 AM: I was able to

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

2021-02-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290613#comment-17290613 ] Chao Sun commented on SPARK-33212: -- I was able to reproduce the error in my local environment, and find

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

2021-02-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290127#comment-17290127 ] Chao Sun commented on SPARK-33212: -- Thanks again [~ouyangxc.zte].

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

2021-02-23 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289652#comment-17289652 ] Chao Sun commented on SPARK-33212: -- Thanks for the details [~ouyangxc.zte]! {quote} Get AMIpFilter

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

2021-02-23 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289200#comment-17289200 ] Chao Sun commented on SPARK-33212: -- Thanks for the report [~ouyangxc.zte]. Can you provide more

[jira] [Created] (SPARK-34419) Move PartitionTransforms from java to scala directory

2021-02-10 Thread Chao Sun (Jira)
Chao Sun created SPARK-34419: Summary: Move PartitionTransforms from java to scala directory Key: SPARK-34419 URL: https://issues.apache.org/jira/browse/SPARK-34419 Project: Spark Issue Type:

[jira] [Created] (SPARK-34347) CatalogImpl.uncacheTable should invalidate in cascade for temp views

2021-02-03 Thread Chao Sun (Jira)
Chao Sun created SPARK-34347: Summary: CatalogImpl.uncacheTable should invalidate in cascade for temp views Key: SPARK-34347 URL: https://issues.apache.org/jira/browse/SPARK-34347 Project: Spark

[jira] [Resolved] (SPARK-34108) Cache lookup doesn't work in certain cases

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-34108. -- Resolution: Duplicate > Cache lookup doesn't work in certain cases >

[jira] [Updated] (SPARK-34108) Cache lookup doesn't work in certain cases

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a temporary or permenant view doesn't work in certain cases. For

[jira] [Updated] (SPARK-34108) Cache lookup doesn't work in certain cases

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Summary: Cache lookup doesn't work in certain cases (was: Caching with permanent view doesn't work in

[jira] [Created] (SPARK-34271) Use majorMinorPatchVersion for Hive version parsing

2021-01-27 Thread Chao Sun (Jira)
Chao Sun created SPARK-34271: Summary: Use majorMinorPatchVersion for Hive version parsing Key: SPARK-34271 URL: https://issues.apache.org/jira/browse/SPARK-34271 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27589) Spark file source V2

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273161#comment-17273161 ] Chao Sun commented on SPARK-27589: -- [~xkrogen] FWIW I'm working on a POC for SPARK-32935 at the moment.

[jira] [Commented] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-26 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272333#comment-17272333 ] Chao Sun commented on SPARK-34052: -- [~hyukjin.kwon] [~cloud_fan] do you think we should include this in

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-19 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268063#comment-17268063 ] Chao Sun commented on SPARK-33507: -- [~aokolnychyi] could you elaborate on the question? currently Spark

[jira] [Comment Edited] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264608#comment-17264608 ] Chao Sun edited comment on SPARK-33507 at 1/14/21, 5:23 AM: Thanks

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264608#comment-17264608 ] Chao Sun commented on SPARK-33507: -- Thanks [~hyukjin.kwon]. From my side, there is no regression.

[jira] [Updated] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Summary: Caching with permanent view doesn't work in certain cases (was: Caching doesn't work

[jira] [Updated] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a permanent view doesn't work in certain cases. For instance, in the

[jira] [Updated] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a permanent view doesn't work in some cases. For instance, in the

[jira] [Created] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-34108: Summary: Caching doesn't work completely with permanent view Key: SPARK-34108 URL: https://issues.apache.org/jira/browse/SPARK-34108 Project: Spark Issue Type:

[jira] [Updated] (SPARK-33311) Improve semantics for REFRESH TABLE

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33311: - Parent: SPARK-33507 Issue Type: Sub-task (was: Improvement) > Improve semantics for REFRESH

[jira] [Updated] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34076: - Affects Version/s: (was: 3.0.1) 3.1.0 > SQLContext.dropTempTable fails if

[jira] [Created] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Chao Sun (Jira)
Chao Sun created SPARK-34076: Summary: SQLContext.dropTempTable fails if cache is non-empty Key: SPARK-34076 URL: https://issues.apache.org/jira/browse/SPARK-34076 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262826#comment-17262826 ] Chao Sun commented on SPARK-33507: -- [~dongjoon], [~hyukjin.kwon]: can either of you remove me as

[jira] [Updated] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33507: - Description: This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark

[jira] [Updated] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33507: - Description: This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark

[jira] [Updated] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > A cached view should become invalid

[jira] [Updated] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Parent: (was: SPARK-33392) Issue Type: Bug (was: Sub-task) > A cached view should become

[jira] [Updated] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-09 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Summary: A cached view should become invalid after a table is dropped (was: A view doesn't become

[jira] [Updated] (SPARK-33729) When refreshing cache, Spark should not use cached plan when recaching data

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33729: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > When refreshing cache, Spark should not

[jira] [Updated] (SPARK-34052) A view doesn't become invalid after a table is dropped or replaced in V2

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Affects Version/s: 3.1.0 > A view doesn't become invalid after a table is dropped or replaced in V2 >

[jira] [Updated] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34039: - Affects Version/s: 3.0.1 > [DSv2] ReplaceTable should invalidate cache >

<    1   2   3   4   5   >