[GitHub] [spark] AmplabJenkins commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-77757 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135046/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-77761 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39630/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.
AmplabJenkins commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-77759 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39631/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
MaxGekk commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775553799 jenkins, retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775552827 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39632/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31496: [SPARK-34384][CORE] API cleanup for ResourceProfile
HyukjinKwon commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-775543510 I don't think blocks RC I am preparing now. Same here too. Hope we can do this earlier next time. @tgravescs, though, some of them like https://github.com/apache/spark/pull/31496/files#diff-ee7e90474f1ce0390fce28f5e4d1d1be689c905ed13069bd869c8689a177e154R150 seems making sense. Maybe it wouldn't hurt to do a fine grained review. For the API changes such as https://github.com/apache/spark/pull/31496/files#diff-a6d96a65d9905b310451b125acac6610ffbd6b4548461bd1d5a18dc29282814aL57, we might have to follow the standard deprecation process if we're going to change if we will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
SparkQA removed a comment on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775536734 **[Test build #135049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775541886 **[Test build #135049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775470550 **[Test build #135046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135046/testReport)** for PR 31493 at commit [`69b2540`](https://github.com/apache/spark/commit/69b25406b9d5ee1f0862f06614f579a4ed9a1e03). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775537415 **[Test build #135046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135046/testReport)** for PR 31493 at commit [`69b2540`](https://github.com/apache/spark/commit/69b25406b9d5ee1f0862f06614f579a4ed9a1e03). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes
SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775537236 **[Test build #135050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135050/testReport)** for PR 31249 at commit [`83c6e2e`](https://github.com/apache/spark/commit/83c6e2edcbf17c07492d102c98ddc623dd0b2110). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775536734 **[Test build #135049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com/apache/spark/commit/ef5d0e99eb42595a5edca5bc4f50258159f8ffe6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.
SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775536471 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39631/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775528864 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39629/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
AmplabJenkins removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775528858 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
AmplabJenkins removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775528860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135043/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775528859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775528864 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39629/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
AmplabJenkins commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775528860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135043/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775390813 **[Test build #135043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135043/testReport)** for PR 31494 at commit [`8b5508b`](https://github.com/apache/spark/commit/8b5508b2625f443362d17b377f6563fdfb897b77). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775526942 **[Test build #135043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135043/testReport)** for PR 31494 at commit [`8b5508b`](https://github.com/apache/spark/commit/8b5508b2625f443362d17b377f6563fdfb897b77). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.
SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775525435 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39631/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775517184 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39628/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775514042 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39629/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775503416 **[Test build #135047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135047/testReport)** for PR 31529 at commit [`1e3e524`](https://github.com/apache/spark/commit/1e3e5246c476ce5b7dce1222b6caf44828984696). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775510620 **[Test build #135047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135047/testReport)** for PR 31529 at commit [`1e3e524`](https://github.com/apache/spark/commit/1e3e5246c476ce5b7dce1222b6caf44828984696). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.
SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775509993 **[Test build #135048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135048/testReport)** for PR 31510 at commit [`bff7afd`](https://github.com/apache/spark/commit/bff7afde412ea15378a5c32d878b6c24a810d24e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request
holdenk commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775509482 LGTM pending CI. Jenkins failure is unrelated but to be safe, Jenkins retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
AmplabJenkins removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775503100 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135044/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775386321 **[Test build #135044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135044/testReport)** for PR 31494 at commit [`5063e4e`](https://github.com/apache/spark/commit/5063e4e76431fb5b37bdc9ed33f6643b9e5cfa33). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775503416 **[Test build #135047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135047/testReport)** for PR 31529 at commit [`1e3e524`](https://github.com/apache/spark/commit/1e3e5246c476ce5b7dce1222b6caf44828984696). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
AmplabJenkins commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775503100 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135044/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775502521 **[Test build #135044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135044/testReport)** for PR 31494 at commit [`5063e4e`](https://github.com/apache/spark/commit/5063e4e76431fb5b37bdc9ed33f6643b9e5cfa33). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31476: [SPARK-34366][SQL] Add interface for DS v2 metrics
viirya commented on a change in pull request #31476:
URL: https://github.com/apache/spark/pull/31476#discussion_r572414083
##
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
##
@@ -102,4 +102,13 @@ default MicroBatchStream toMicroBatchStream(String
checkpointLocation) {
default ContinuousStream toContinuousStream(String checkpointLocation) {
throw new UnsupportedOperationException(description() + ": Continuous scan
are not supported");
}
+
+ /**
+ * Returns an array of supported custom metrics with name and description.
+ * By default it returns empty array.
+ */
+ default CustomMetric[] supportedCustomMetrics() {
Review comment:
cc @dongjoon-hyun Does it convincing for you?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775497614 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39629/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes
holdenk commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775491530 Thanks @Ngone51 :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on pull request #31475: [SPARK-34360][SQL] Support table truncation by v2 Table Catalogs
rdblue commented on pull request #31475: URL: https://github.com/apache/spark/pull/31475#issuecomment-775490751 @MaxGekk, can you share the use case that you have for this? You mentioned truncation-specific optimizations. I think working with concrete use cases is usually a good idea. If these are theoretical only -- like a user that can drop all data but not a subset -- then we should put this off. If there's a specific case, then let's discuss it. I agree that there _may_ be good reason to pass that the engine's intent was to truncate. That's why we have `SupportsTruncate` for the write builder. And I agree with you that we don't necessarily need to use an atomic operation that could truncate and add data at the same time. Your point about not having insert permissions is a good one to justify not using `SupportsTruncate`, although the case of a user that can drop all data but not subsets doesn't sound real. The point about truncation possibly being a metadata operation is why we added `SupportsDelete` at the table level. Those points may indicate that an interface to truncate a table as a stand-alone operation is valid, although I still think that it is a bad idea to add more interfaces to v2 without a reasonable expectation that they will actually be used. Another problem here is that this is operation is proposed at the catalog level, which does not fit with how v2 works. I think that the reason for this is emulating what the Hive does, but that's not usually a good choice. In v2, catalogs load tables and tables are modified. That's why `SupportsDelete` extends `Table` and not `TableCatalog`. This keeps concerns separate, so we have a way to handle tables that don't exist and a separate way to handle tables that don't support a certain operation. Mixing those two together at the catalog level over-complicates the API, requiring a source to throw one exception if the table doesn't exist and another if it doesn't support truncation. (We also went through this discussion with the recently added interfaces to add/drop partitions.) Assuming that it is worth adding this interface, I would expect it to be a mix-in for `Table`. And like `SupportsOverwrite` that implements `SupportsTruncate`, I think this should also update `SupportsDelete` so that tables don't need to implement both interfaces. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.
srowen commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775483346 Jenkins test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map
srowen commented on pull request #31484: URL: https://github.com/apache/spark/pull/31484#issuecomment-775482881 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map
srowen closed pull request #31484: URL: https://github.com/apache/spark/pull/31484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31527: [MINOR] Add a note about pip installation test in RC for release vote template
dongjoon-hyun commented on pull request #31527: URL: https://github.com/apache/spark/pull/31527#issuecomment-775482765 +1, late LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
sunchao commented on a change in pull request #31524: URL: https://github.com/apache/spark/pull/31524#discussion_r572383191 ## File path: docs/sql-ref-syntax-ddl-drop-table.md ## @@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e In case of an external table, only the associated metadata information is removed from the metastore database. +If the table is cached, the command uncaches the table and all its dependants such as views. Thereby the dependants should be cached again explicitly. Review comment: nit: dependants -> dependents ## File path: docs/sql-ref-syntax-ddl-alter-view.md ## @@ -28,6 +28,8 @@ the name of a view to a different name, set and unset the metadata of the view b Renames the existing view. If the new view name already exists in the source database, a `TableAlreadyExistsException` is thrown. This operation does not support moving the views across databases. +If the view is cached, the command clears cached data of the view and all its dependents that refer to it. View's cache will be lazily filled when the next time the view or the dependents are accessed. The command leaves view's dependents as uncached. Review comment: Hmm is this correct? should we say "The view's cache will be lazily filled when the next time it is accessed"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
AmplabJenkins removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775468802 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135045/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775468806 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135042/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
AmplabJenkins removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775468804 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map
AmplabJenkins removed a comment on pull request #31484: URL: https://github.com/apache/spark/pull/31484#issuecomment-775468801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135039/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775470550 **[Test build #135046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135046/testReport)** for PR 31493 at commit [`69b2540`](https://github.com/apache/spark/commit/69b25406b9d5ee1f0862f06614f579a4ed9a1e03). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775469874 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39628/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775468806 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135042/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map
AmplabJenkins commented on pull request #31484: URL: https://github.com/apache/spark/pull/31484#issuecomment-775468801 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135039/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
AmplabJenkins commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775468804 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775468802 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135045/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775355318 **[Test build #135042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135042/testReport)** for PR 31493 at commit [`77f970d`](https://github.com/apache/spark/commit/77f970d72d09335d0d92a9dadf24e03693a60b58). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775457133 **[Test build #135042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135042/testReport)** for PR 31493 at commit [`77f970d`](https://github.com/apache/spark/commit/77f970d72d09335d0d92a9dadf24e03693a60b58). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775452893 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39627/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775436704 **[Test build #135045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135045/testReport)** for PR 31529 at commit [`79a8a59`](https://github.com/apache/spark/commit/79a8a597bec72778b9ad73fe0336d42548cad51e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775449509 **[Test build #135045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135045/testReport)** for PR 31529 at commit [`79a8a59`](https://github.com/apache/spark/commit/79a8a597bec72778b9ad73fe0336d42548cad51e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map
SparkQA removed a comment on pull request #31484: URL: https://github.com/apache/spark/pull/31484#issuecomment-775264831 **[Test build #135039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135039/testReport)** for PR 31484 at commit [`7eac600`](https://github.com/apache/spark/commit/7eac600ab39d9a9463c1945f92702ee575a0ee95). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map
SparkQA commented on pull request #31484: URL: https://github.com/apache/spark/pull/31484#issuecomment-775440637 **[Test build #135039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135039/testReport)** for PR 31484 at commit [`7eac600`](https://github.com/apache/spark/commit/7eac600ab39d9a9463c1945f92702ee575a0ee95). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
imback82 commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572353356
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala
##
@@ -1141,6 +1141,36 @@ trait AlterTableTests extends SharedSparkSession {
}
}
+ test("AlterTable: remove nonexistent table property") {
+val t = s"${catalogAndNamespace}table_name"
+withTable(t) {
+ sql(s"CREATE TABLE $t (id int) USING $v2Format TBLPROPERTIES('test' =
'34')")
+
+ val tableName = fullTableName(t)
+ val table = getTableMetadata(tableName)
+
+ assert(table.name === tableName)
+ assert(table.properties ===
+withDefaultOwnership(Map("provider" -> v2Format, "test" ->
"34")).asJava)
+
+ val exc = intercept[AnalysisException] {
+sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('unknown')")
+ }
+ assert(exc.getMessage.contains("Attempted to unset non-existent property
'unknown'"))
+
+ // Reserved property "comment" should be allowed regardless.
Review comment:
Yes, that's the current behavior of V1 command:
https://github.com/apache/spark/blob/3b26bc25362a245a610c3e222b971b4ae612bc3e/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L307-L314
Since table comment is optional, it will just be set to `None` if the table
had no comment:
https://github.com/apache/spark/blob/3b26bc25362a245a610c3e222b971b4ae612bc3e/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L316
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775436704 **[Test build #135045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135045/testReport)** for PR 31529 at commit [`79a8a59`](https://github.com/apache/spark/commit/79a8a597bec72778b9ad73fe0336d42548cad51e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
MaxGekk edited a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775424148 @cloud-fan @gengliangwang @HyukjinKwon Could you review this PR, please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31462: [SPARK-34347][SQL] CatalogImpl.uncacheTable should invalidate in cascade for temp views
sunchao commented on pull request #31462: URL: https://github.com/apache/spark/pull/31462#issuecomment-775429482 Any more comments on this @viirya ? cc @dongjoon-hyun too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
MaxGekk commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775424148 @cloud-fan @HyukjinKwon Could you review this PR, please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31378: [SPARK-34240][SQL] Unify output of `SHOW TBLPROPERTIES` clause's output attribute's schema and ExprID
AmplabJenkins removed a comment on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-775419237 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135037/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
AmplabJenkins removed a comment on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775419235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775420690 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39627/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31378: [SPARK-34240][SQL] Unify output of `SHOW TBLPROPERTIES` clause's output attribute's schema and ExprID
AmplabJenkins commented on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-775419237 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135037/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
AmplabJenkins commented on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775419236 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk opened a new pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read
MaxGekk opened a new pull request #31529:
URL: https://github.com/apache/spark/pull/31529
### What changes were proposed in this pull request?
In the PR, I propose new option `datetimeRebaseMode` for the Avro
datasource. The option influences on loading ancient dates and timestamps
column values from avro files.
The option supports the same values as the SQL config
`spark.sql.legacy.avro.datetimeRebaseModeInRead` namely;
- `"LEGACY"`, when an option is set to this value, Spark rebases
dates/timestamps from the legacy hybrid calendar (Julian + Gregorian) to the
Proleptic Gregorian calendar.
- `"CORRECTED"`, dates/timestamps are read AS IS from avro files.
- `"EXCEPTION"`, when it is set as an option value, Spark will fail the
reading if it sees ancient dates/timestamps that are ambiguous between the two
calendars.
### Why are the changes needed?
1. New options will allow to load avro files from at least two sources in
different rebasing modes in the same query. For instance:
```scala
val df1 = spark.read.option("datetimeRebaseMode",
"legacy").format("avro").load(folder1)
val df2 = spark.read.option("datetimeRebaseMode",
"corrected").format("avro").load(folder2)
df1.join(df2, ...)
```
Before the changes, it is impossible because the SQL config
`spark.sql.legacy.avro.datetimeRebaseModeInRead` influences on both reads.
2. Mixing of Dataset/DataFrame and RDD APIs should become possible. Since
SQL configs are not propagated through RDDs, the following code fails on
ancient timestamps:
```scala
spark.conf.set("spark.sql.legacy.avro.datetimeRebaseModeInRead", "legacy")
spark.read.format("avro").load(folder).distinct.rdd.collect()
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
By running the modified test suites:
```
$ build/sbt "test:testOnly *AvroV1Suite"
$ build/sbt "test:testOnly *AvroV2Suite"
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
cloud-fan commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572333847
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala
##
@@ -1141,6 +1141,36 @@ trait AlterTableTests extends SharedSparkSession {
}
}
+ test("AlterTable: remove nonexistent table property") {
+val t = s"${catalogAndNamespace}table_name"
+withTable(t) {
+ sql(s"CREATE TABLE $t (id int) USING $v2Format TBLPROPERTIES('test' =
'34')")
+
+ val tableName = fullTableName(t)
+ val table = getTableMetadata(tableName)
+
+ assert(table.name === tableName)
+ assert(table.properties ===
+withDefaultOwnership(Map("provider" -> v2Format, "test" ->
"34")).asJava)
+
+ val exc = intercept[AnalysisException] {
+sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('unknown')")
+ }
+ assert(exc.getMessage.contains("Attempted to unset non-existent property
'unknown'"))
+
+ // Reserved property "comment" should be allowed regardless.
Review comment:
even if the table has no comment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
cloud-fan commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r57265
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala
##
@@ -1141,6 +1141,36 @@ trait AlterTableTests extends SharedSparkSession {
}
}
+ test("AlterTable: remove nonexistent table property") {
+val t = s"${catalogAndNamespace}table_name"
+withTable(t) {
+ sql(s"CREATE TABLE $t (id int) USING $v2Format TBLPROPERTIES('test' =
'34')")
+
+ val tableName = fullTableName(t)
+ val table = getTableMetadata(tableName)
+
+ assert(table.name === tableName)
+ assert(table.properties ===
+withDefaultOwnership(Map("provider" -> v2Format, "test" ->
"34")).asJava)
+
+ val exc = intercept[AnalysisException] {
+sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('unknown')")
+ }
+ assert(exc.getMessage.contains("Attempted to unset non-existent property
'unknown'"))
+
+ // Reserved property "comment" should be allowed regardless.
+ sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('comment')")
Review comment:
ah, I see!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
SparkQA commented on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775401954 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39624/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes
holdenk commented on a change in pull request #31249:
URL: https://github.com/apache/spark/pull/31249#discussion_r572326065
##
File path:
core/src/test/scala/org/apache/spark/scheduler/HealthTrackerSuite.scala
##
@@ -554,6 +554,50 @@ class HealthTrackerSuite extends SparkFunSuite with
BeforeAndAfterEach with Mock
verify(allocationClientMock).killExecutorsOnHost("hostA")
}
+ test("excluding decommission and kills executors when enabled") {
+val allocationClientMock = mock[ExecutorAllocationClient]
+
+// verify we decommission when configured
+conf.set(config.EXCLUDE_ON_FAILURE_KILL_ENABLED, true)
+conf.set(config.DECOMMISSION_ENABLED.key, "true")
+conf.set(config.EXCLUDE_ON_FAILURE_DECOMMISSION_ENABLED.key, "true")
+conf.set(config.MAX_FAILURES_PER_EXEC.key, "1")
+conf.set(config.MAX_FAILED_EXEC_PER_NODE.key, "2")
+healthTracker = new HealthTracker(listenerBusMock, conf,
Some(allocationClientMock), clock)
+
+// Fail 4 tasks in one task set on executor 1, so that executor gets
excluded for the whole
+// application.
+val taskSetExclude2 = createTaskSetExcludelist(stageId = 0)
+(0 until 4).foreach { partition =>
+ taskSetExclude2.updateExcludedForFailedTask(
+"hostA", exec = "1", index = partition, failureReason = "testing")
+}
+healthTracker.updateExcludedForSuccessfulTaskSet(0, 0,
taskSetExclude2.execToFailures)
+
+val msg1 =
+ "Killing excluded executor id 1 since
spark.excludeOnFailure.killExcludedExecutors is set."
Review comment:
sure
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31378: [SPARK-34240][SQL] Unify output of `SHOW TBLPROPERTIES` clause's output attribute's schema and ExprID
SparkQA removed a comment on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-775221906 **[Test build #135037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135037/testReport)** for PR 31378 at commit [`cc9d8ac`](https://github.com/apache/spark/commit/cc9d8ac1105a336997384eab475fc82458d92b01). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31378: [SPARK-34240][SQL] Unify output of `SHOW TBLPROPERTIES` clause's output attribute's schema and ExprID
SparkQA commented on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-775398487 **[Test build #135037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135037/testReport)** for PR 31378 at commit [`cc9d8ac`](https://github.com/apache/spark/commit/cc9d8ac1105a336997384eab475fc82458d92b01). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes
holdenk commented on a change in pull request #31249:
URL: https://github.com/apache/spark/pull/31249#discussion_r572321583
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1942,6 +1950,15 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createOptional
+ private[spark] val EXECUTOR_DECOMMISSION_CLEANUP_INTERVAL =
+ConfigBuilder("spark.executor.decommission.cleanupInterval")
Review comment:
Yeah sure that sounds like a reasonable change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
SparkQA removed a comment on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775231660 **[Test build #135035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135035/testReport)** for PR 31485 at commit [`0285c88`](https://github.com/apache/spark/commit/0285c88c3ee9d9bb9988cfd2c3627e27a52b2f8d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
SparkQA commented on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775392322 **[Test build #135035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135035/testReport)** for PR 31485 at commit [`0285c88`](https://github.com/apache/spark/commit/0285c88c3ee9d9bb9988cfd2c3627e27a52b2f8d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775390813 **[Test build #135043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135043/testReport)** for PR 31494 at commit [`8b5508b`](https://github.com/apache/spark/commit/8b5508b2625f443362d17b377f6563fdfb897b77). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775386321 **[Test build #135044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135044/testReport)** for PR 31494 at commit [`5063e4e`](https://github.com/apache/spark/commit/5063e4e76431fb5b37bdc9ed33f6643b9e5cfa33). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775380295 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39625/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31258: [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning
AmplabJenkins removed a comment on pull request #31258: URL: https://github.com/apache/spark/pull/31258#issuecomment-775380293 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135033/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
AmplabJenkins removed a comment on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775380294 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
imback82 commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572305204
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala
##
@@ -1141,6 +1141,36 @@ trait AlterTableTests extends SharedSparkSession {
}
}
+ test("AlterTable: remove nonexistent table property") {
+val t = s"${catalogAndNamespace}table_name"
+withTable(t) {
+ sql(s"CREATE TABLE $t (id int) USING $v2Format TBLPROPERTIES('test' =
'34')")
+
+ val tableName = fullTableName(t)
+ val table = getTableMetadata(tableName)
+
+ assert(table.name === tableName)
+ assert(table.properties ===
+withDefaultOwnership(Map("provider" -> v2Format, "test" ->
"34")).asJava)
+
+ val exc = intercept[AnalysisException] {
+sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('unknown')")
+ }
+ assert(exc.getMessage.contains("Attempted to unset non-existent property
'unknown'"))
+
+ // Reserved property "comment" should be allowed regardless.
+ sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('comment')")
Review comment:
The "comment" property seems to be treated differently that other
reserved properties (not checked in the following):
https://github.com/apache/spark/blob/3b26bc25362a245a610c3e222b971b4ae612bc3e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L2836-L2850
The following works:
```
scala> sql("CREATE TABLE t (id int) TBLPROPERTIES('comment' = 'hello')")
scala> sql("DESCRIBE EXTENDED t").show
+++---+
|col_name| data_type|comment|
+++---+
| id| int| null|
||| |
|# Detailed Table ...|| |
|Database| default| |
| Table| t| |
| Owner| terryk| |
|Created Time|Mon Feb 08 11:08:...| |
| Last Access| UNKNOWN| |
| Created By| Spark 3.0.1| |
|Type| MANAGED| |
|Provider|hive| |
| Comment| hello| |
```
Note the following to see how v1 command can unset the comment using this
command:
https://github.com/apache/spark/blob/3b26bc25362a245a610c3e222b971b4ae612bc3e/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L315-L316
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31258: [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning
AmplabJenkins commented on pull request #31258: URL: https://github.com/apache/spark/pull/31258#issuecomment-775380293 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135033/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
AmplabJenkins commented on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775380294 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
AmplabJenkins commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775380295 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39625/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
SparkQA removed a comment on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775315610 **[Test build #135041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135041/testReport)** for PR 31485 at commit [`29916c8`](https://github.com/apache/spark/commit/29916c899795997befc8ad8944200ceeea8b5a7d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats
SparkQA commented on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775377002 **[Test build #135041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135041/testReport)** for PR 31485 at commit [`29916c8`](https://github.com/apache/spark/commit/29916c899795997befc8ad8944200ceeea8b5a7d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31476: [SPARK-34366][SQL] Add interface for DS v2 metrics
cloud-fan commented on a change in pull request #31476:
URL: https://github.com/apache/spark/pull/31476#discussion_r572300669
##
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/SupportsReportMetrics.java
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * Data sources can implement this interface to
+ * report supported custom metrics to Spark in read/write path.
+ *
+ * @since 3.2.0
+ */
+@Evolving
+public interface SupportsReportMetrics {
+
+/**
+ * Returns an array of supported custom metrics with name and description.
+ * By default it returns empty array.
+ */
+default CustomMetric[] supportedCustomMetrics() {
+CustomMetric[] NO_METRICS = {};
Review comment:
shall we define it as a private static field?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775376682 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39625/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31476: [SPARK-34366][SQL] Add interface for DS v2 metrics
cloud-fan commented on a change in pull request #31476:
URL: https://github.com/apache/spark/pull/31476#discussion_r572298710
##
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
##
@@ -102,4 +102,13 @@ default MicroBatchStream toMicroBatchStream(String
checkpointLocation) {
default ContinuousStream toContinuousStream(String checkpointLocation) {
throw new UnsupportedOperationException(description() + ": Continuous scan
are not supported");
}
+
+ /**
+ * Returns an array of supported custom metrics with name and description.
+ * By default it returns empty array.
+ */
+ default CustomMetric[] supportedCustomMetrics() {
Review comment:
I don't see how a mixin trait helps avoid breaking change. Removing the
trait later is also a breaking change. Agree with @rdblue that an optional
method is simple and sufficient. Some other features have mixin trait because
they don't have a reasonable default if made into optional methods. Metrics is
OK as we can return empty metrics as the default.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks
SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775374134 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39625/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31475: [SPARK-34360][SQL] Support table truncation by v2 Table Catalogs
MaxGekk commented on pull request #31475: URL: https://github.com/apache/spark/pull/31475#issuecomment-775373033 > ... why is this necessary instead of deleting from the table or overwriting everything with no new records? 1. By emulating table truncation via the insertion of no rows, you require atomic operations: delete + insert but a concrete implementation might not support this though it can atomically truncate a table. 2. You close the room for truncation specific optimizations. If a catalog implementation would know in advance that we want to truncate the entire table instead of deleting all rows, it could do that in a more optimal way. Let's say some file based implementation could move the table folder to a trash folder using one atomic syscall. 3. From security or permissions controls point of view, we could distinguish insert with overwrite (or delete) from truncation. I could imagine a case when some roles/users can have only truncation permissions but not insert or delete permissions. 4. Also it is possible that truncation op is just a record at catalog level log but inserts/deletes are records at table level logs. So, we cannot smoothly sit on such implementation if we emulate table truncation via inserts/deletes. In general, I do believe we should not hide our intention from catalog implementations - truncation should be explicit. Table catalog implementation should decide how to implement in a more optimal way. So, if they can emulate truncation via overwriting with no rows, ok, this is up to them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
cloud-fan commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572293623
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala
##
@@ -1141,6 +1141,36 @@ trait AlterTableTests extends SharedSparkSession {
}
}
+ test("AlterTable: remove nonexistent table property") {
+val t = s"${catalogAndNamespace}table_name"
+withTable(t) {
+ sql(s"CREATE TABLE $t (id int) USING $v2Format TBLPROPERTIES('test' =
'34')")
+
+ val tableName = fullTableName(t)
+ val table = getTableMetadata(tableName)
+
+ assert(table.name === tableName)
+ assert(table.properties ===
+withDefaultOwnership(Map("provider" -> v2Format, "test" ->
"34")).asJava)
+
+ val exc = intercept[AnalysisException] {
+sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('unknown')")
+ }
+ assert(exc.getMessage.contains("Attempted to unset non-existent property
'unknown'"))
+
+ // Reserved property "comment" should be allowed regardless.
+ sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('comment')")
Review comment:
hmm, IIRC we don't even allow users to set the reserved property in
CREATE TABLE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
cloud-fan commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572292880
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableProperties.scala
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.plans.logical.{AlterTableUnsetProperties,
LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.connector.catalog.TableCatalog
+
+/**
+ * A rule for resolving table properties such as handling non-existent
properties.
Review comment:
`... resolving AlterTableUnsetProperties such as ...`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
cloud-fan commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572292383
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableProperties.scala
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.plans.logical.{AlterTableUnsetProperties,
LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.connector.catalog.TableCatalog
+
+/**
+ * A rule for resolving table properties such as handling non-existent
properties.
+ */
+object ResolveTableProperties extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
+case a @ AlterTableUnsetProperties(r: ResolvedTable, props, ifExists) if
!ifExists =>
Review comment:
nvm, it's possible that someone else adds the table property before this
command gets actually executed. It's safer to always execute this command.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
imback82 commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572291602
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableProperties.scala
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.plans.logical.{AlterTableUnsetProperties,
LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.connector.catalog.TableCatalog
+
+/**
+ * A rule for resolving table properties such as handling non-existent
properties.
+ */
+object ResolveTableProperties extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
+case a @ AlterTableUnsetProperties(r: ResolvedTable, props, ifExists) if
!ifExists =>
+ val tblProperties = r.table.properties.asScala
+ props.foreach { p =>
+if (!tblProperties.contains(p) && p != TableCatalog.PROP_COMMENT) {
+ throw new AnalysisException(
+s"Attempted to unset non-existent property '$p' in table
'${r.identifier.quoted}'")
Review comment:
Note that we cannot remove the logic in
https://github.com/apache/spark/blob/e614f34c7a538b1f2c59616689eaea95af85fd54/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L307-L311
because `ALTER VIEW` still needs it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command
cloud-fan commented on a change in pull request #31494:
URL: https://github.com/apache/spark/pull/31494#discussion_r572291464
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableProperties.scala
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.plans.logical.{AlterTableUnsetProperties,
LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.connector.catalog.TableCatalog
+
+/**
+ * A rule for resolving table properties such as handling non-existent
properties.
+ */
+object ResolveTableProperties extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
+case a @ AlterTableUnsetProperties(r: ResolvedTable, props, ifExists) if
!ifExists =>
Review comment:
Can we turn it into `NoopCommand` if possible?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
