[GitHub] [spark] LuciferYang commented on pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
LuciferYang commented on pull request #33748: URL: https://github.com/apache/spark/pull/33748#issuecomment-899274159 There will be some duplicate codes in the two PR, and this part of the code will be synchronized after one of them is merged -- This is an automated message from the

[GitHub] [spark] cloud-fan commented on a change in pull request #33737: [SPARK-36508][SQL] ANSI type coercion: disallow binary operations between Interval and String literal

2021-08-16 Thread GitBox
cloud-fan commented on a change in pull request #33737: URL: https://github.com/apache/spark/pull/33737#discussion_r689285715 ## File path: sql/core/src/test/resources/sql-tests/results/interval.sql.out ## @@ -2175,6 +2289,54 @@ org.apache.spark.sql.AnalysisException cannot

[GitHub] [spark] MaxGekk closed pull request #33742: [SPARK-36491][SQL] Make from_json/to_json to handle timestamp_ntz type properly

2021-08-16 Thread GitBox
MaxGekk closed pull request #33742: URL: https://github.com/apache/spark/pull/33742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] MaxGekk commented on pull request #33742: [SPARK-36491][SQL] Make from_json/to_json to handle timestamp_ntz type properly

2021-08-16 Thread GitBox
MaxGekk commented on pull request #33742: URL: https://github.com/apache/spark/pull/33742#issuecomment-899272375 +1, LGTM. Merging to master. Thank you, @sarutak . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] LuciferYang edited a comment on pull request #30483: [SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2021-08-16 Thread GitBox
LuciferYang edited a comment on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-899271543 @dongjoon-hyun Because https://github.com/apache/spark/pull/33748 gives an ORC-only pr and use a new JIRA SPARK-36516, I'll change this PR to Parquet-only

[GitHub] [spark] LuciferYang commented on pull request #30483: [SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2021-08-16 Thread GitBox
LuciferYang commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-899271543 @dongjoon-hyun Because https://github.com/apache/spark/pull/33748 gives an ORC-only pr, I'll change this PR to Parquet-only -- This is an automated message from

[GitHub] [spark] SparkQA commented on pull request #32467: [WIP] simplify correlated subquery resolution

2021-08-16 Thread GitBox
SparkQA commented on pull request #32467: URL: https://github.com/apache/spark/pull/32467#issuecomment-899271081 **[Test build #142486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142486/testReport)** for PR 32467 at commit

[GitHub] [spark] SparkQA commented on pull request #33673: [SPARK-36448][SQL] Exceptions in NoSuchItemException.scala have to be case classes

2021-08-16 Thread GitBox
SparkQA commented on pull request #33673: URL: https://github.com/apache/spark/pull/33673#issuecomment-899269965 **[Test build #142479 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142479/testReport)** for PR 33673 at commit

[GitHub] [spark] SparkQA commented on pull request #32475: [SPARK-34775][SQL] Push down limit through window when partitionSpec is not empty

2021-08-16 Thread GitBox
SparkQA commented on pull request #32475: URL: https://github.com/apache/spark/pull/32475#issuecomment-899269428 **[Test build #142483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142483/testReport)** for PR 32475 at commit

[GitHub] [spark] SparkQA commented on pull request #28032: [SPARK-31264][SQL] Repartition before writing data source tables/directories

2021-08-16 Thread GitBox
SparkQA commented on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-899268787 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46989/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
SparkQA commented on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899266804 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46986/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
SparkQA commented on pull request #33748: URL: https://github.com/apache/spark/pull/33748#issuecomment-899264134 **[Test build #142496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142496/testReport)** for PR 33748 at commit

[GitHub] [spark] LuciferYang commented on a change in pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
LuciferYang commented on a change in pull request #33748: URL: https://github.com/apache/spark/pull/33748#discussion_r689274315 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileMeta.scala ## @@ -0,0 +1,49 @@ +/* + * Licensed to the

[GitHub] [spark] AmplabJenkins commented on pull request #32468: [SPARK-35335][SQL] Coalesce shuffle partition as much as possible for REPARTITION_BY_NONE

2021-08-16 Thread GitBox
AmplabJenkins commented on pull request #32468: URL: https://github.com/apache/spark/pull/32468#issuecomment-899262904 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142485/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
AmplabJenkins commented on pull request #33748: URL: https://github.com/apache/spark/pull/33748#issuecomment-899262527 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142495/ -- This

[GitHub] [spark] SparkQA commented on pull request #32468: [SPARK-35335][SQL] Coalesce shuffle partition as much as possible for REPARTITION_BY_NONE

2021-08-16 Thread GitBox
SparkQA commented on pull request #32468: URL: https://github.com/apache/spark/pull/32468#issuecomment-899262446 **[Test build #142485 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142485/testReport)** for PR 32468 at commit

[GitHub] [spark] SparkQA commented on pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
SparkQA commented on pull request #33748: URL: https://github.com/apache/spark/pull/33748#issuecomment-899262512 **[Test build #142495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142495/testReport)** for PR 33748 at commit

[GitHub] [spark] SparkQA commented on pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
SparkQA commented on pull request #33748: URL: https://github.com/apache/spark/pull/33748#issuecomment-899261804 **[Test build #142495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142495/testReport)** for PR 33748 at commit

[GitHub] [spark] SparkQA commented on pull request #30483: [SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2021-08-16 Thread GitBox
SparkQA commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-899260866 **[Test build #142494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142494/testReport)** for PR 30483 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30483: [SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2021-08-16 Thread GitBox
AmplabJenkins commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-899260894 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142494/ -- This

[GitHub] [spark] SparkQA commented on pull request #30483: [SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

2021-08-16 Thread GitBox
SparkQA commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-899258205 **[Test build #142494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142494/testReport)** for PR 30483 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition

2021-08-16 Thread GitBox
AmplabJenkins removed a comment on pull request #28642: URL: https://github.com/apache/spark/pull/28642#issuecomment-895866583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] LuciferYang opened a new pull request #33748: [SPARK-36516][SQL] Add File Metadata cache support for Orc

2021-08-16 Thread GitBox
LuciferYang opened a new pull request #33748: URL: https://github.com/apache/spark/pull/33748 ### What changes were proposed in this pull request? The main purpose of this pr is to introduce the File Meta Cache mechanism for Spark SQL and the basic File Meta Cache implementation for Orc

[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull from join condition

2021-08-16 Thread GitBox
SparkQA commented on pull request #28642: URL: https://github.com/apache/spark/pull/28642#issuecomment-899257491 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46988/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #33175: [SPARK-35973][SQL] DataSourceV2: Support SHOW CATALOGS

2021-08-16 Thread GitBox
AmplabJenkins commented on pull request #33175: URL: https://github.com/apache/spark/pull/33175#issuecomment-899255953 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan closed pull request #32467: [WIP] simplify correlated subquery resolution

2021-08-16 Thread GitBox
cloud-fan closed pull request #32467: URL: https://github.com/apache/spark/pull/32467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
AmplabJenkins removed a comment on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899254332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33683: [SPARK-36041][SS][DOCS] Introduce the RocksDBStateStoreProvider in the programming guide

2021-08-16 Thread GitBox
AmplabJenkins removed a comment on pull request #33683: URL: https://github.com/apache/spark/pull/33683#issuecomment-899254331 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46987/

[GitHub] [spark] AmplabJenkins commented on pull request #33683: [SPARK-36041][SS][DOCS] Introduce the RocksDBStateStoreProvider in the programming guide

2021-08-16 Thread GitBox
AmplabJenkins commented on pull request #33683: URL: https://github.com/apache/spark/pull/33683#issuecomment-899254331 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46987/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
AmplabJenkins commented on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899254332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33683: [SPARK-36041][SS][DOCS] Introduce the RocksDBStateStoreProvider in the programming guide

2021-08-16 Thread GitBox
AmplabJenkins removed a comment on pull request #33683: URL: https://github.com/apache/spark/pull/33683#issuecomment-899233222 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142488/

[GitHub] [spark] SparkQA commented on pull request #33683: [SPARK-36041][SS][DOCS] Introduce the RocksDBStateStoreProvider in the programming guide

2021-08-16 Thread GitBox
SparkQA commented on pull request #33683: URL: https://github.com/apache/spark/pull/33683#issuecomment-899253895 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46987/ --

[GitHub] [spark] SparkQA commented on pull request #28032: [SPARK-31264][SQL] Repartition before writing data source tables/directories

2021-08-16 Thread GitBox
SparkQA commented on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-899253827 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46989/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
SparkQA removed a comment on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899171541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
SparkQA commented on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899250993 **[Test build #142477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142477/testReport)** for PR 33736 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33683: [SPARK-36041][SS][DOCS] Introduce the RocksDBStateStoreProvider in the programming guide

2021-08-16 Thread GitBox
SparkQA removed a comment on pull request #33683: URL: https://github.com/apache/spark/pull/33683#issuecomment-899224072 **[Test build #142488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142488/testReport)** for PR 33683 at commit

[GitHub] [spark] SparkQA commented on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
SparkQA commented on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899250123 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46986/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33736: [SPARK-35991][SQL] Add PlanStability suite for TPCH

2021-08-16 Thread GitBox
SparkQA commented on pull request #33736: URL: https://github.com/apache/spark/pull/33736#issuecomment-899244825 **[Test build #142476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142476/testReport)** for PR 33736 at commit

<    2   3   4   5   6   7