[jira] [Created] (SPARK-32802) Avoid using SpecificInternalRow in RunLengthEncoding#Encoder

2020-09-04 Thread Chao Sun (Jira)
Chao Sun created SPARK-32802: Summary: Avoid using SpecificInternalRow in RunLengthEncoding#Encoder Key: SPARK-32802 URL: https://issues.apache.org/jira/browse/SPARK-32802 Project: Spark Issue

[jira] [Updated] (SPARK-24994) Support cast pushdown for integral types

2020-09-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-24994: - Summary: Support cast pushdown for integral types (was: Support pushdown casts for ) > Support cast

[jira] [Updated] (SPARK-24994) Support pushdown casts for

2020-09-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-24994: - Summary: Support pushdown casts for (was: When the data type of the field is converted to other

[jira] [Created] (SPARK-32858) UnwrapCastInBinaryComparison: support more types other than Integral

2020-09-11 Thread Chao Sun (Jira)
Chao Sun created SPARK-32858: Summary: UnwrapCastInBinaryComparison: support more types other than Integral Key: SPARK-32858 URL: https://issues.apache.org/jira/browse/SPARK-32858 Project: Spark

[jira] [Created] (SPARK-32880) Improve and refactor parallel listing in HadoopFSUtils

2020-09-14 Thread Chao Sun (Jira)
Chao Sun created SPARK-32880: Summary: Improve and refactor parallel listing in HadoopFSUtils Key: SPARK-32880 URL: https://issues.apache.org/jira/browse/SPARK-32880 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

2020-08-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187849#comment-17187849 ] Chao Sun commented on SPARK-27733: -- bq. BTW, is Hive community will to introduce the version upgrade of

[jira] [Created] (SPARK-33135) Use listLocatedStatus from FileSystem implementations

2020-10-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-33135: Summary: Use listLocatedStatus from FileSystem implementations Key: SPARK-33135 URL: https://issues.apache.org/jira/browse/SPARK-33135 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

2020-10-12 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212804#comment-17212804 ] Chao Sun commented on SPARK-27733: -- [~iemejia] I can help with Hive releases if you can come up with

[jira] [Commented] (SPARK-32694) Pushdown cast to data sources

2020-08-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184614#comment-17184614 ] Chao Sun commented on SPARK-32694: -- Thanks [~rakson] for the pointer! didn't know there were multiple

[jira] [Created] (SPARK-32703) Re-enable dictionary filtering for Parquet

2020-08-25 Thread Chao Sun (Jira)
Chao Sun created SPARK-32703: Summary: Re-enable dictionary filtering for Parquet Key: SPARK-32703 URL: https://issues.apache.org/jira/browse/SPARK-32703 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32703: - Summary: Enable dictionary filtering for Parquet vectorized reader (was: Re-enable dictionary

[jira] [Updated] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32703: - Description: Parquet vectorized reader still uses the old API for {{filterRowGroups}} and only filters

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

2020-08-26 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185293#comment-17185293 ] Chao Sun commented on SPARK-27733: -- bq. Apache Spark 3.0 will ship both Hive 1.2.1 built-in and Hive

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to:

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to:

[jira] [Created] (SPARK-32674) Add suggestion for parallel directory listing in tuning doc

2020-08-20 Thread Chao Sun (Jira)
Chao Sun created SPARK-32674: Summary: Add suggestion for parallel directory listing in tuning doc Key: SPARK-32674 URL: https://issues.apache.org/jira/browse/SPARK-32674 Project: Spark Issue

[jira] [Created] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
Chao Sun created SPARK-32721: Summary: Simplify if clauses with null and boolean Key: SPARK-32721 URL: https://issues.apache.org/jira/browse/SPARK-32721 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to:

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to:

[jira] [Updated] (SPARK-32694) Pushdown cast to data sources

2020-08-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32694: - Description: Currently we don't support pushing down cast to data source (see

[jira] [Created] (SPARK-32694) Pushdown cast to data sources

2020-08-24 Thread Chao Sun (Jira)
Chao Sun created SPARK-32694: Summary: Pushdown cast to data sources Key: SPARK-32694 URL: https://issues.apache.org/jira/browse/SPARK-32694 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32694) Pushdown cast to data sources

2020-08-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32694: - Description: Currently we don't support pushing down cast to data source (see

[jira] [Commented] (SPARK-32694) Pushdown cast to data sources

2020-08-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183592#comment-17183592 ] Chao Sun commented on SPARK-32694: -- Seems there could be more cases like this, such as: {code} scala>

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

2020-09-23 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201094#comment-17201094 ] Chao Sun commented on SPARK-27733: -- [~sha...@uber.com] sure, I can join in the next sync meeting. >

[jira] [Updated] (SPARK-32858) UnwrapCastInBinaryComparison: support other numeric types

2020-09-17 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32858: - Summary: UnwrapCastInBinaryComparison: support other numeric types (was: UnwrapCastInBinaryComparison:

[jira] [Updated] (SPARK-29250) Upgrade to Hadoop 3.2.2

2020-10-21 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-29250: - Summary: Upgrade to Hadoop 3.2.2 (was: Upgrade to Hadoop 3.2.1 and move to shaded client) > Upgrade

[jira] [Created] (SPARK-33212) Move to shaded clients for Hadoop 3.x profile

2020-10-21 Thread Chao Sun (Jira)
Chao Sun created SPARK-33212: Summary: Move to shaded clients for Hadoop 3.x profile Key: SPARK-33212 URL: https://issues.apache.org/jira/browse/SPARK-33212 Project: Spark Issue Type:

[jira] [Created] (SPARK-33213) Upgrade Apache Arrow to 2.0.0

2020-10-21 Thread Chao Sun (Jira)
Chao Sun created SPARK-33213: Summary: Upgrade Apache Arrow to 2.0.0 Key: SPARK-33213 URL: https://issues.apache.org/jira/browse/SPARK-33213 Project: Spark Issue Type: Dependency upgrade

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262826#comment-17262826 ] Chao Sun commented on SPARK-33507: -- [~dongjoon], [~hyukjin.kwon]: can either of you remove me as

[jira] [Updated] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34076: - Affects Version/s: (was: 3.0.1) 3.1.0 > SQLContext.dropTempTable fails if

[jira] [Created] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Chao Sun (Jira)
Chao Sun created SPARK-34076: Summary: SQLContext.dropTempTable fails if cache is non-empty Key: SPARK-34076 URL: https://issues.apache.org/jira/browse/SPARK-34076 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-06 Thread Chao Sun (Jira)
Chao Sun created SPARK-34039: Summary: [DSv2] ReplaceTable should invalidate cache Key: SPARK-34039 URL: https://issues.apache.org/jira/browse/SPARK-34039 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-06 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260280#comment-17260280 ] Chao Sun commented on SPARK-34039: -- I'll submit a patch for this soon. > [DSv2] ReplaceTable should

[jira] [Created] (SPARK-33567) DSv2: Use callback instead of passing Spark session and v2 relation for refreshing cache

2020-11-25 Thread Chao Sun (Jira)
Chao Sun created SPARK-33567: Summary: DSv2: Use callback instead of passing Spark session and v2 relation for refreshing cache Key: SPARK-33567 URL: https://issues.apache.org/jira/browse/SPARK-33567

[jira] [Created] (SPARK-33652) DSv2: DeleteFrom should refresh cache

2020-12-03 Thread Chao Sun (Jira)
Chao Sun created SPARK-33652: Summary: DSv2: DeleteFrom should refresh cache Key: SPARK-33652 URL: https://issues.apache.org/jira/browse/SPARK-33652 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-33652) DSv2: DeleteFrom should refresh cache

2020-12-03 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33652: - Parent: SPARK-33507 Issue Type: Sub-task (was: Improvement) > DSv2: DeleteFrom should refresh

[jira] [Created] (SPARK-33653) DSv2: REFRESH TABLE should recache the table itself

2020-12-03 Thread Chao Sun (Jira)
Chao Sun created SPARK-33653: Summary: DSv2: REFRESH TABLE should recache the table itself Key: SPARK-33653 URL: https://issues.apache.org/jira/browse/SPARK-33653 Project: Spark Issue Type:

[jira] [Created] (SPARK-33729) When refreshing cache, Spark should not use cached plan when recaching data

2020-12-09 Thread Chao Sun (Jira)
Chao Sun created SPARK-33729: Summary: When refreshing cache, Spark should not use cached plan when recaching data Key: SPARK-33729 URL: https://issues.apache.org/jira/browse/SPARK-33729 Project: Spark

[jira] [Created] (SPARK-33880) Replace Hadoop profile names by using hadoop-2 and hadoop-3

2020-12-22 Thread Chao Sun (Jira)
Chao Sun created SPARK-33880: Summary: Replace Hadoop profile names by using hadoop-2 and hadoop-3 Key: SPARK-33880 URL: https://issues.apache.org/jira/browse/SPARK-33880 Project: Spark Issue

[jira] [Commented] (SPARK-33213) Upgrade Apache Arrow to 2.0.0

2020-11-09 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228892#comment-17228892 ] Chao Sun commented on SPARK-33213: -- Thanks. I think there are still a few improvements on the Java side

[jira] [Created] (SPARK-33435) DSv2: REFRESH TABLE should invalidate caches

2020-11-12 Thread Chao Sun (Jira)
Chao Sun created SPARK-33435: Summary: DSv2: REFRESH TABLE should invalidate caches Key: SPARK-33435 URL: https://issues.apache.org/jira/browse/SPARK-33435 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-33437) Support CACHE TABLE command in DSv2

2020-11-12 Thread Chao Sun (Jira)
Chao Sun created SPARK-33437: Summary: Support CACHE TABLE command in DSv2 Key: SPARK-33437 URL: https://issues.apache.org/jira/browse/SPARK-33437 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-33437) Support CACHE TABLE command in DSv2

2020-11-12 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33437: - Issue Type: New Feature (was: Improvement) > Support CACHE TABLE command in DSv2 >

[jira] [Updated] (SPARK-33435) DSv2: REFRESH TABLE should invalidate caches

2020-11-12 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33435: - Labels: DSv2 (was: ) > DSv2: REFRESH TABLE should invalidate caches >

[jira] [Updated] (SPARK-33435) DSv2: REFRESH TABLE should invalidate caches

2020-11-12 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33435: - Environment: (was: R) > DSv2: REFRESH TABLE should invalidate caches >

[jira] [Updated] (SPARK-33493) Scala 2.13 build with SBT failure

2020-11-19 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33493: - Component/s: SQL > Scala 2.13 build with SBT failure > - > >

[jira] [Created] (SPARK-33493) Scala 2.13 build with SBT failure

2020-11-19 Thread Chao Sun (Jira)
Chao Sun created SPARK-33493: Summary: Scala 2.13 build with SBT failure Key: SPARK-33493 URL: https://issues.apache.org/jira/browse/SPARK-33493 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-33492) DSv2: Append/Overwrite/ReplaceTable should invalidate cache

2020-11-19 Thread Chao Sun (Jira)
Chao Sun created SPARK-33492: Summary: DSv2: Append/Overwrite/ReplaceTable should invalidate cache Key: SPARK-33492 URL: https://issues.apache.org/jira/browse/SPARK-33492 Project: Spark Issue

[jira] [Created] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2020-11-20 Thread Chao Sun (Jira)
Chao Sun created SPARK-33507: Summary: Improve and fix cache behavior in v1 and v2 Key: SPARK-33507 URL: https://issues.apache.org/jira/browse/SPARK-33507 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-33435) DSv2: REFRESH TABLE should invalidate caches

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33435: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > DSv2: REFRESH TABLE should invalidate

[jira] [Updated] (SPARK-33492) DSv2: Append/Overwrite/ReplaceTable should invalidate cache

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33492: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > DSv2: Append/Overwrite/ReplaceTable

[jira] [Updated] (SPARK-33305) DSv2: DROP TABLE command should also invalidate cache

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33305: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > DSv2: DROP TABLE command should also

[jira] [Updated] (SPARK-33290) REFRESH TABLE should invalidate cache even though the table itself may not be cached

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33290: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > REFRESH TABLE should invalidate cache

[jira] [Updated] (SPARK-33305) DSv2: DROP TABLE command should also invalidate cache

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33305: - Parent: (was: SPARK-33392) Issue Type: Bug (was: Sub-task) > DSv2: DROP TABLE command

[jira] [Updated] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33507: - Description: This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2020-11-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236521#comment-17236521 ] Chao Sun commented on SPARK-33507: -- [~dongjoon] Yes this is for Spark 3.1 _mostly_ (some JIRAs are in

[jira] [Created] (SPARK-33290) REFRESH TABLE should invalidate cache even though the table itself may not be cached

2020-10-29 Thread Chao Sun (Jira)
Chao Sun created SPARK-33290: Summary: REFRESH TABLE should invalidate cache even though the table itself may not be cached Key: SPARK-33290 URL: https://issues.apache.org/jira/browse/SPARK-33290

[jira] [Created] (SPARK-33305) DSv2: DROP TABLE command should also invalidate cache

2020-10-30 Thread Chao Sun (Jira)
Chao Sun created SPARK-33305: Summary: DSv2: DROP TABLE command should also invalidate cache Key: SPARK-33305 URL: https://issues.apache.org/jira/browse/SPARK-33305 Project: Spark Issue Type:

[jira] [Updated] (SPARK-33290) REFRESH TABLE should invalidate cache even though the table itself may not be cached

2020-10-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33290: - Affects Version/s: 2.4.7 > REFRESH TABLE should invalidate cache even though the table itself may not

[jira] [Created] (SPARK-33311) Improve semantics for REFRESH TABLE

2020-10-31 Thread Chao Sun (Jira)
Chao Sun created SPARK-33311: Summary: Improve semantics for REFRESH TABLE Key: SPARK-33311 URL: https://issues.apache.org/jira/browse/SPARK-33311 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-33311) Improve semantics for REFRESH TABLE

2020-10-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33311: - Description: Currently, the semantics for {{REFRESH TABLE t}} is not well defined for view (let's say

[jira] [Updated] (SPARK-33311) Improve semantics for REFRESH TABLE

2020-10-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33311: - Description: Currently, the semantics for {{REFRESH TABLE t}} is not well defined for view (let's say

[jira] [Updated] (SPARK-29250) Upgrade to Hadoop 3.2.1 and move to shaded client

2020-10-20 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-29250: - Summary: Upgrade to Hadoop 3.2.1 and move to shaded client (was: Upgrade to Hadoop 3.2.1 Upgrade to

[jira] [Updated] (SPARK-33212) Move to shaded clients for Hadoop 3.x profile

2020-10-22 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33212: - Description: Hadoop 3.x+ offers shaded client jars: hadoop-client-api and hadoop-client-runtime, which

[jira] [Updated] (SPARK-33212) Move to shaded clients for Hadoop 3.x profile

2020-10-22 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33212: - Description: Hadoop 3.x+ offers shaded client jars: hadoop-client-api and hadoop-client-runtime, which

[jira] [Updated] (SPARK-33212) Move to shaded clients for Hadoop 3.x profile

2020-10-22 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33212: - Description: Hadoop 3.x+ offers shaded client jars: hadoop-client-api and hadoop-client-runtime, which

[jira] [Updated] (SPARK-29250) Upgrade to Hadoop 3.3.0

2020-10-22 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-29250: - Summary: Upgrade to Hadoop 3.3.0 (was: Upgrade to Hadoop 3.2.2) > Upgrade to Hadoop 3.3.0 >

[jira] [Created] (SPARK-33293) Refactor WriteToDataSourceV2Exec and reduce code duplication

2020-10-29 Thread Chao Sun (Jira)
Chao Sun created SPARK-33293: Summary: Refactor WriteToDataSourceV2Exec and reduce code duplication Key: SPARK-33293 URL: https://issues.apache.org/jira/browse/SPARK-33293 Project: Spark Issue

[jira] [Created] (SPARK-33376) Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader

2020-11-06 Thread Chao Sun (Jira)
Chao Sun created SPARK-33376: Summary: Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader Key: SPARK-33376 URL: https://issues.apache.org/jira/browse/SPARK-33376 Project: Spark

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261036#comment-17261036 ] Chao Sun commented on SPARK-34039: -- I'm not sure if this is a bug since: 1. the command is only

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261468#comment-17261468 ] Chao Sun commented on SPARK-34039: -- Thanks for verifying this [~dongjoon]! > [DSv2] ReplaceTable

[jira] [Updated] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34039: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > [DSv2] ReplaceTable should invalidate

[jira] [Updated] (SPARK-34052) A view doesn't become invalid after a table is dropped or replaced in V2

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Affects Version/s: 3.1.0 > A view doesn't become invalid after a table is dropped or replaced in V2 >

[jira] [Updated] (SPARK-33729) When refreshing cache, Spark should not use cached plan when recaching data

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33729: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > When refreshing cache, Spark should not

[jira] [Created] (SPARK-34052) A view doesn't become invalid after a table is dropped or replaced in V2

2021-01-08 Thread Chao Sun (Jira)
Chao Sun created SPARK-34052: Summary: A view doesn't become invalid after a table is dropped or replaced in V2 Key: SPARK-34052 URL: https://issues.apache.org/jira/browse/SPARK-34052 Project: Spark

[jira] [Updated] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34039: - Affects Version/s: 3.0.1 > [DSv2] ReplaceTable should invalidate cache >

[jira] [Updated] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-09 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Summary: A cached view should become invalid after a table is dropped (was: A view doesn't become

[jira] [Updated] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Parent: (was: SPARK-33392) Issue Type: Bug (was: Sub-task) > A cached view should become

[jira] [Updated] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34052: - Parent: SPARK-33507 Issue Type: Sub-task (was: Bug) > A cached view should become invalid

[jira] [Updated] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33507: - Description: This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark

[jira] [Updated] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33507: - Description: This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-19 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268063#comment-17268063 ] Chao Sun commented on SPARK-33507: -- [~aokolnychyi] could you elaborate on the question? currently Spark

[jira] [Updated] (SPARK-33311) Improve semantics for REFRESH TABLE

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33311: - Parent: SPARK-33507 Issue Type: Sub-task (was: Improvement) > Improve semantics for REFRESH

[jira] [Updated] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a permanent view doesn't work in some cases. For instance, in the

[jira] [Created] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-34108: Summary: Caching doesn't work completely with permanent view Key: SPARK-34108 URL: https://issues.apache.org/jira/browse/SPARK-34108 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a permanent view doesn't work in certain cases. For instance, in the

[jira] [Updated] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Summary: Caching with permanent view doesn't work in certain cases (was: Caching doesn't work

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264608#comment-17264608 ] Chao Sun commented on SPARK-33507: -- Thanks [~hyukjin.kwon]. From my side, there is no regression.

[jira] [Comment Edited] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264608#comment-17264608 ] Chao Sun edited comment on SPARK-33507 at 1/14/21, 5:23 AM: Thanks

[jira] [Commented] (SPARK-33937) Move the old partition data to trash instead of deleting it when inserting rewrite hive table

2020-12-29 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256192#comment-17256192 ] Chao Sun commented on SPARK-33937: -- This looks like a duplicate of SPARK-32480. > Move the old

[jira] [Created] (SPARK-34347) CatalogImpl.uncacheTable should invalidate in cascade for temp views

2021-02-03 Thread Chao Sun (Jira)
Chao Sun created SPARK-34347: Summary: CatalogImpl.uncacheTable should invalidate in cascade for temp views Key: SPARK-34347 URL: https://issues.apache.org/jira/browse/SPARK-34347 Project: Spark

[jira] [Commented] (SPARK-34052) A cached view should become invalid after a table is dropped

2021-01-26 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272333#comment-17272333 ] Chao Sun commented on SPARK-34052: -- [~hyukjin.kwon] [~cloud_fan] do you think we should include this in

[jira] [Resolved] (SPARK-34108) Cache lookup doesn't work in certain cases

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-34108. -- Resolution: Duplicate > Cache lookup doesn't work in certain cases >

[jira] [Updated] (SPARK-34108) Cache lookup doesn't work in certain cases

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a temporary or permenant view doesn't work in certain cases. For

[jira] [Updated] (SPARK-34108) Cache lookup doesn't work in certain cases

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Summary: Cache lookup doesn't work in certain cases (was: Caching with permanent view doesn't work in

[jira] [Created] (SPARK-34271) Use majorMinorPatchVersion for Hive version parsing

2021-01-27 Thread Chao Sun (Jira)
Chao Sun created SPARK-34271: Summary: Use majorMinorPatchVersion for Hive version parsing Key: SPARK-34271 URL: https://issues.apache.org/jira/browse/SPARK-34271 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27589) Spark file source V2

2021-01-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273161#comment-17273161 ] Chao Sun commented on SPARK-27589: -- [~xkrogen] FWIW I'm working on a POC for SPARK-32935 at the moment.

[jira] [Created] (SPARK-35743) Improve Parquet vectorized reader

2021-06-11 Thread Chao Sun (Jira)
Chao Sun created SPARK-35743: Summary: Improve Parquet vectorized reader Key: SPARK-35743 URL: https://issues.apache.org/jira/browse/SPARK-35743 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35640: - Parent: SPARK-35743 Issue Type: Sub-task (was: Improvement) > Refactor Parquet vectorized

[jira] [Commented] (SPARK-34861) Support nested column in Spark vectorized readers

2021-06-10 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361090#comment-17361090 ] Chao Sun commented on SPARK-34861: -- Synced with [~chengsu] offline and I will take over this JIRA. >

  1   2   3   4   5   >