[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522346027 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522346029 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14367/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522346027 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522346029 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14367/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
gengliangwang commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522345748 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522344141 **[Test build #109299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109299/testReport)** for PR 25443 at commit [`9defec2`](https://github.com/apache/spark/commit/9defec229199e690a5b25734ba3159182b382e3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522343977 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522343979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14366/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522343977 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522343979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14366/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522343868 I triggered the test since the test abortion looks weird. ``` org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite *** ABORTED *** java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522343773 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522342914 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109294/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522342913 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522342914 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109294/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522342913 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
SparkQA removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522318080 **[Test build #109294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109294/testReport)** for PR 25443 at commit [`9defec2`](https://github.com/apache/spark/commit/9defec229199e690a5b25734ba3159182b382e3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522342837 **[Test build #109294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109294/testReport)** for PR 25443 at commit [`9defec2`](https://github.com/apache/spark/commit/9defec229199e690a5b25734ba3159182b382e3c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522342414 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109297/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
SparkQA removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522340475 **[Test build #109297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109297/testReport)** for PR 25461 at commit [`721c4f2`](https://github.com/apache/spark/commit/721c4f2a3c4d215f2d8545456c6d8c861454b1ba). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522342412 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522342394 **[Test build #109297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109297/testReport)** for PR 25461 at commit [`721c4f2`](https://github.com/apache/spark/commit/721c4f2a3c4d215f2d8545456c6d8c861454b1ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522342414 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109297/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522342412 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-522340859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14364/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-522340857 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
SparkQA commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-522340992 **[Test build #109298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109298/testReport)** for PR 25487 at commit [`bc5042b`](https://github.com/apache/spark/commit/bc5042b13a2fc779f485bccc3b85d080a05d026c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522340852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14365/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522340848 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522340852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14365/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-522340859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14364/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-522340857 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522340848 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak opened a new pull request #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
sarutak opened a new pull request #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487 ### What changes were proposed in this pull request? Improved warning message in Barrier Execution Mode when required slots > maximum slots. The new message contains information about required slots, maximum slots and how many times retry failed. ### Why are the changes needed? Providing to users with the number of required slots, maximum slots and how many times retry failed might help users to decide what they should do. For example, continuing to wait for retry succeeded or killing jobs. ### Does this PR introduce any user-facing change? Yes. If `spark.scheduler.barrier.maxConcurrentTaskCheck.maxFailures=3`, we get following warning message. Before applying this change: ``` 19/08/18 15:18:09 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently. 19/08/18 15:18:24 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently. 19/08/18 15:18:39 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently. 19/08/18 15:18:54 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently. org.apache.spark.scheduler.BarrierJobSlotsNumberCheckFailed: [SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently. Please init a new cluster with more CPU cores or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage. at org.apache.spark.scheduler.DAGScheduler.checkBarrierStageWithNumSlots(DAGScheduler.scala:439) at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:453) at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:983) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2140) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2132) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2121) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:749) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2080) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2145) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:961) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:366) at org.apache.spark.rdd.RDD.collect(RDD.scala:960) ... 47 elided ``` After applying this change: ``` 19/08/18 16:52:23 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently. 19/08/18 16:52:38 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently (Retry 1/3 failed). 19/08/18 16:52:53 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently (Retry 2/3 failed). 19/08/18 16:53:08 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently (Retry 3/3 failed). org.apache.spark.scheduler.BarrierJobSlotsNumberCheckFailed: [SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently. Please init a new cluster with more CPU cores or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage. at org.apache.spark.scheduler.DAGScheduler.checkBarrierStageWithNumSlots(DAGScheduler.scala:439) at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:453) at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:983) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2140) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2132) at
[GitHub] [spark] SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-522340475 **[Test build #109297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109297/testReport)** for PR 25461 at commit [`721c4f2`](https://github.com/apache/spark/commit/721c4f2a3c4d215f2d8545456c6d8c861454b1ba). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
gengliangwang commented on a change in pull request #25461:
[SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314994323
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -1279,8 +1409,26 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
(c, evPrim, evNull) => code"$evNull = true;"
case TimestampType =>
(c, evPrim, evNull) => code"$evPrim = (long)
${timestampToIntegerCode(c)};"
+case DecimalType() if failOnIntegerOverflow =>
+ (c, evPrim, evNull) =>
+code"""
+ try {
+$evPrim = $c.toJavaBigInteger().longValueExact();
+ } catch (ArithmeticException e) {
+throw new ArithmeticException("Casting $c to Long causes
overflow");
+ }
+"""
case DecimalType() =>
(c, evPrim, evNull) => code"$evPrim = $c.toLong();"
+case _: FloatType | _: DoubleType if failOnIntegerOverflow =>
+ (c, evPrim, evNull) =>
+code"""
+ if ($c >= ${Long.MinValue}L && $c <= ${Long.MaxValue}L) {
Review comment:
I am not sure about that. Keeping the current behavior seems fine as well,
since we have printed the exact value which causes casting failure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
gengliangwang commented on a change in pull request #25461:
[SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314994248
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -1075,4 +1075,113 @@ class CastSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkEvaluation(cast("badvalue", dataType), null)
}
}
+
+ private def testIntMaxAndMin(dt: DataType): Unit = {
+Seq(Int.MaxValue + 1L, Int.MinValue - 1L).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, dt),
"overflow")
+
checkExceptionInExpression[ArithmeticException](cast(Decimal(value.toString),
dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast(Literal(value * MICROS_PER_SECOND, TimestampType), dt),
"overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast(Literal(value * 1.5f, FloatType), dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast(Literal(value * 1.0, DoubleType), dt), "overflow")
+}
+ }
+
+ private def testLongMaxAndMin(dt: DataType): Unit = {
+Seq(Decimal(Long.MaxValue) + Decimal(1), Decimal(Long.MinValue) -
Decimal(1)).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](
+cast(value, dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast((value * Decimal(1.1)).toFloat, dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast((value * Decimal(1.1)).toDouble, dt), "overflow")
+}
+ }
+
+ test("Cast to byte with option FAIL_ON_INTEGER_OVERFLOW enabled") {
+withSQLConf(SQLConf.FAIL_ON_INTEGER_OVERFLOW.key -> "true") {
+ testIntMaxAndMin(ByteType)
+ Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
+checkExceptionInExpression[ArithmeticException](cast(value, ByteType),
"overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType),
"overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
+ }
+
+ Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
+checkEvaluation(cast(value, ByteType), value)
+checkEvaluation(cast(value.toString, ByteType), value)
+checkEvaluation(cast(Decimal(value.toString), ByteType), value)
+checkEvaluation(cast(Literal(value * MICROS_PER_SECOND,
TimestampType), ByteType), value)
+checkEvaluation(cast(Literal(value.toInt, DateType), ByteType), null)
+checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType),
value)
+checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType),
value)
+ }
+}
+ }
+
+ test("Cast to short with option FAIL_ON_INTEGER_OVERFLOW enabled") {
+withSQLConf(SQLConf.FAIL_ON_INTEGER_OVERFLOW.key -> "true") {
+ testIntMaxAndMin(ShortType)
+ Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
+checkExceptionInExpression[ArithmeticException](cast(value,
ShortType), "overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType),
"overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
+ }
+
+ Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
+checkEvaluation(cast(value, ShortType), value)
+checkEvaluation(cast(value.toString, ShortType), value)
+checkEvaluation(cast(Decimal(value.toString), ShortType), value)
+checkEvaluation(cast(Literal(value * MICROS_PER_SECOND,
TimestampType), ShortType), value)
+checkEvaluation(cast(Literal(value.toInt, DateType), ShortType), null)
+checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType),
value)
+checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType),
value)
+ }
+}
+ }
+
+ test("Cast to int with option FAIL_ON_INTEGER_OVERFLOW enabled") {
+withSQLConf(SQLConf.FAIL_ON_INTEGER_OVERFLOW.key -> "true") {
+ testIntMaxAndMin(IntegerType)
+ testLongMaxAndMin(IntegerType)
+
+ Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
+checkEvaluation(cast(value, IntegerType), value)
+checkEvaluation(cast(value.toString, IntegerType), value)
+checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
+
[GitHub] [spark] gengliangwang commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
gengliangwang commented on a change in pull request #25461:
[SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314993178
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -494,7 +506,27 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
case DateType =>
buildCast[Int](_, d => null)
case TimestampType =>
- buildCast[Long](_, t => timestampToLong(t).toShort)
+ buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (!failOnIntegerOverflow || longValue == longValue.toShort) {
+ longValue.toShort
+} else {
+ throw new ArithmeticException(s"Casting $t to Short causes
overflow.")
Review comment:
I was following the error message from
https://github.com/apache/spark/pull/21599.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
gengliangwang commented on a change in pull request #25461:
[SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314993133
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -494,7 +506,27 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
case DateType =>
buildCast[Int](_, d => null)
case TimestampType =>
- buildCast[Long](_, t => timestampToLong(t).toShort)
+ buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (!failOnIntegerOverflow || longValue == longValue.toShort) {
+ longValue.toShort
+} else {
+ throw new ArithmeticException(s"Casting $t to Short causes
overflow.")
+}
+ })
+case x: NumericType if failOnIntegerOverflow =>
+ b =>
+val intValue = try {
+ x.exactNumeric.asInstanceOf[Numeric[Any]].toInt(b)
Review comment:
Well, we can do it by match it case by case. Then the code is a bit long.
Casting to short/byte should be minor usage. Also, The previous code also cast
to `Int` before cast to `Short`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shivusondur commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP] Update the TODO with actual blocking JIRA IDs
shivusondur commented on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP] Update the TODO with actual blocking JIRA IDs URL: https://github.com/apache/spark/pull/25415#issuecomment-522335220 > Yea that's fine to update comments. But @shivusondur can you confirm if you are unable to fix any test or some tests to have used in GROUP BY clause due to both JIRAs? If you can, let's add some tests and only comment out the other tests not working by both JIRAs. @HyukjinKwon >There are 3 instances of groupby test in udf-select_having.sql, all 3 are not working due to the same reason. > Originally we have copied this "udf-select_having.sql" from "select_having.sql", so we are maintaining the same tests as original, (https://github.com/apache/spark/pull/25161#discussion_r305665269) > if the extra tests we can add in new file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522333444 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522333444 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522333445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109293/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522333445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109293/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522317427 **[Test build #109293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109293/testReport)** for PR 24829 at commit [`ad5363b`](https://github.com/apache/spark/commit/ad5363b0f02ff7bce73a7bbff31f171bb3bb2192). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522333292 **[Test build #109293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109293/testReport)** for PR 24829 at commit [`ad5363b`](https://github.com/apache/spark/commit/ad5363b0f02ff7bce73a7bbff31f171bb3bb2192). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25486: [MINOR][SQL] Make analysis error msg more meaningful on DISTINCT queries
dongjoon-hyun closed pull request #25486: [MINOR][SQL] Make analysis error msg more meaningful on DISTINCT queries URL: https://github.com/apache/spark/pull/25486 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful
AmplabJenkins commented on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful URL: https://github.com/apache/spark/pull/25486#issuecomment-522331673 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful
AmplabJenkins commented on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful URL: https://github.com/apache/spark/pull/25486#issuecomment-522331675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful
AmplabJenkins removed a comment on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful URL: https://github.com/apache/spark/pull/25486#issuecomment-522331673 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful
AmplabJenkins removed a comment on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful URL: https://github.com/apache/spark/pull/25486#issuecomment-522331675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful
SparkQA commented on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful URL: https://github.com/apache/spark/pull/25486#issuecomment-522331518 **[Test build #109292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109292/testReport)** for PR 25486 at commit [`065c538`](https://github.com/apache/spark/commit/065c538f3074371e62443f545e6d862ced896f24). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful
SparkQA removed a comment on issue #25486: [MINOR][SQL] Make analysis error messages more meaningful URL: https://github.com/apache/spark/pull/25486#issuecomment-522315610 **[Test build #109292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109292/testReport)** for PR 25486 at commit [`065c538`](https://github.com/apache/spark/commit/065c538f3074371e62443f545e6d862ced896f24). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #22018: [SPARK-25038][SQL] Get block location in parallel
wangyum commented on issue #22018: [SPARK-25038][SQL] Get block location in parallel URL: https://github.com/apache/spark/pull/22018#issuecomment-522329759 Hi @habren I think this performance issue has been fixed by [SPARK-27801](https://issues.apache.org/jira/browse/SPARK-27801). Could you close this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522329333 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109295/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522329331 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522329282 **[Test build #109295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109295/testReport)** for PR 25423 at commit [`bae6524`](https://github.com/apache/spark/commit/bae6524d7d1965dbf30eb4146c1278e18af13801). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
SparkQA removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522319180 **[Test build #109295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109295/testReport)** for PR 25423 at commit [`bae6524`](https://github.com/apache/spark/commit/bae6524d7d1965dbf30eb4146c1278e18af13801). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522329333 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109295/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522329331 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
SparkQA commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-522328696 **[Test build #109296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109296/testReport)** for PR 23531 at commit [`5dc624d`](https://github.com/apache/spark/commit/5dc624d7835bc15686364f564a0d017b5604f7d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-522328534 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14363/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-522328532 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-522328532 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-522328534 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14363/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables
dongjoon-hyun edited a comment on issue #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables URL: https://github.com/apache/spark/pull/22502#issuecomment-522327173 Hi, guys. As @cloud-fan mentioned, since there is a regression case, I'll revert this from `branch-2.4` and `branch-2.3` for `2.4.4/2.3.4` release to prevent VOTE failures. In the master branch, we need to use those test case in #24715 . cc @kiszk since he is a release manager for 2.3.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables
dongjoon-hyun edited a comment on issue #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables URL: https://github.com/apache/spark/pull/22502#issuecomment-522327173 Hi, guys. As @cloud-fan mentioned, there is a regression case. I'll revert this from `branch-2.4` and `branch-2.3` for `2.4.4/2.3.4` release to prevent VOTE failures. In the master branch, we need to use those test case in #24715 . cc @kiszk since he is a release manager for 2.3.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables
dongjoon-hyun commented on issue #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables URL: https://github.com/apache/spark/pull/22502#issuecomment-522327173 Hi, guys. As @cloud-fan mentioned, there is a regression case. I'll revert this from `branch-2.4` and `branch-2.3` for `2.4.4/2.3.4` release. In the master branch, we need to use those test case in #24715 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25484: [SPARK-28647][WEBUI][2.4] Recover additional metric feature
dongjoon-hyun closed pull request #25484: [SPARK-28647][WEBUI][2.4] Recover additional metric feature URL: https://github.com/apache/spark/pull/25484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
HyukjinKwon edited a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522324073 If we turn to `warn`, then we won't have to change a lot of instances here, Yes. But one concern is though then it will show 1000 lines of warnings about the style. One way to workaround this 1000 lines warnings, we could disable this rule when we actually build, linter-check (see for instance https://github.com/apache/spark/commit/761fc46c70f15ffc3f35ff87b6a5c9192af1c55f). The Intellij actually looks useful to avoid excessive (and useless) string interpolation, which people still nitpick in PRs time to time. So, in this way, we could be able to do the scalatyle check only in IntelliJ and does not affect anything like build or existing codes. But not sure if guys here like this idea given the complexity, and .. 3.0.0 might be the last candidate to do this .. maybe at least with few years. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
HyukjinKwon edited a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522324073 If we turn to `warn`, then we won't have to change a lot of instances here, Yes. But one concern is though then it will show 1000 lines of warnings about the style. One way to workaround this 1000 lines warnings, we could disable this rule when we actually build, linter-check (see for instance https://github.com/apache/spark/commit/761fc46c70f15ffc3f35ff87b6a5c9192af1c55f). The Intellij actually looks useful to avoid excessive (and useless) string interpolation, which people still nitpick in PRs time to time. So, in this way, we could be able to the style only in IntelliJ and does not affect anything like build or existing codes. But not sure if guys here like this idea given the complexity, and .. 3.0.0 might be the last candidate to do this .. maybe at least with few years. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
HyukjinKwon commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522324073 If we turn to `warn`, then we won't have to change a lot of instances here, Yes. But one concern is though then it will show 1000 lines of warnings about the style. One way to workaround this 1000 lines warnings, we could disable this rule when we actually build (see for instance https://github.com/apache/spark/commit/761fc46c70f15ffc3f35ff87b6a5c9192af1c55f). The Intellij actually looks useful to avoid excessive (and useless) string interpolation, which people still nitpick in PRs time to time. So, in this way, we could be able to the style only in IntelliJ and does not affect anything like build or existing codes. But not sure if guys here like this idea given the complexity, and .. 3.0.0 might be the last candidate to do this .. maybe at least with few years. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base
HyukjinKwon commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base URL: https://github.com/apache/spark/pull/25371#issuecomment-522322651 Looks fine. Let me take another look tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base
HyukjinKwon commented on a change in pull request #25371:
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF
test base
URL: https://github.com/apache/spark/pull/25371#discussion_r314985850
##
File path: sql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-join.sql
##
@@ -0,0 +1,2081 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- JOIN
+-- Test JOIN clauses
+--
https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/join.sql
+--
+-- This test file was converted from pgSQL/join.sql.
+
+CREATE OR REPLACE TEMPORARY VIEW INT4_TBL AS SELECT * FROM
+ (VALUES (0), (123456), (-123456), (2147483647), (-2147483647))
+ AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW INT8_TBL AS SELECT * FROM
+ (VALUES
+(123, 456),
+(123, 4567890123456789),
+(4567890123456789, 123),
+(4567890123456789, 4567890123456789),
+(4567890123456789, -4567890123456789))
+ AS v(q1, q2);
+CREATE OR REPLACE TEMPORARY VIEW FLOAT8_TBL AS SELECT * FROM
+ (VALUES (0.0), (1004.30), (-34.84),
+(cast('1.2345678901234e+200' as double)), (cast('1.2345678901234e-200' as
double)))
+ AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW TEXT_TBL AS SELECT * FROM
+ (VALUES ('doh!'), ('hi de ho neighbor'))
+ AS v(f1);
+CREATE OR REPLACE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1;
+
+CREATE TABLE J1_TBL (
+ i integer,
+ j integer,
+ t string
+) USING parquet;
+
+CREATE TABLE J2_TBL (
+ i integer,
+ k integer
+) USING parquet;
+
+
+INSERT INTO J1_TBL VALUES (1, 4, 'one');
+INSERT INTO J1_TBL VALUES (2, 3, 'two');
+INSERT INTO J1_TBL VALUES (3, 2, 'three');
+INSERT INTO J1_TBL VALUES (4, 1, 'four');
+INSERT INTO J1_TBL VALUES (5, 0, 'five');
+INSERT INTO J1_TBL VALUES (6, 6, 'six');
+INSERT INTO J1_TBL VALUES (7, 7, 'seven');
+INSERT INTO J1_TBL VALUES (8, 8, 'eight');
+INSERT INTO J1_TBL VALUES (0, NULL, 'zero');
+INSERT INTO J1_TBL VALUES (NULL, NULL, 'null');
+INSERT INTO J1_TBL VALUES (NULL, 0, 'zero');
+
+INSERT INTO J2_TBL VALUES (1, -1);
+INSERT INTO J2_TBL VALUES (2, 2);
+INSERT INTO J2_TBL VALUES (3, -3);
+INSERT INTO J2_TBL VALUES (2, 4);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (5, -5);
+INSERT INTO J2_TBL VALUES (0, NULL);
+INSERT INTO J2_TBL VALUES (NULL, NULL);
+INSERT INTO J2_TBL VALUES (NULL, 0);
+
+-- [SPARK-20856] Do not need onerow because it only used for test statement
using nested joins
+-- useful in some tests below
+-- create temp table onerow();
+-- insert into onerow default values;
+-- analyze onerow;
+
+
+--
+-- CORRELATION NAMES
+-- Make sure that table/column aliases are supported
+-- before diving into more complex join syntax.
+--
+
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
+ FROM J1_TBL AS tx;
+
+SELECT udf(udf('')) AS `xxx`, udf(udf(i)), udf(j), udf(t)
+ FROM J1_TBL tx;
+
+SELECT udf('') AS `xxx`, a, udf(udf(b)), c
+ FROM J1_TBL AS t1 (a, b, c);
+
+SELECT udf('') AS `xxx`, udf(a), udf(b), udf(udf(c))
+ FROM J1_TBL t1 (a, b, c);
+
+SELECT udf('') AS `xxx`, udf(a), b, udf(c), udf(d), e
+ FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e);
+
+-- [SPARK-28377] Fully support correlation names in the FROM clause
+-- SELECT '' AS "xxx", t1.a, t2.e
+-- FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e)
+-- WHERE t1.a = t2.d;
+
+
+--
+-- CROSS JOIN
+-- Qualifications are not allowed on cross joins,
+-- which degenerate into a standard unqualified inner join.
+--
+
+SELECT udf('') AS `xxx`, *
+ FROM J1_TBL CROSS JOIN J2_TBL;
+
+-- ambiguous column
+SELECT udf('') AS `xxx`, udf(i), udf(k), udf(t)
+ FROM J1_TBL CROSS JOIN J2_TBL;
+
+-- resolve previous ambiguity by specifying the table name
+SELECT udf('') AS `xxx`, udf(t1.i), udf(k), udf(t)
+ FROM J1_TBL t1 CROSS JOIN J2_TBL t2;
+
+SELECT udf(udf('')) AS `xxx`, udf(udf(ii)), udf(udf(tt)), udf(udf(kk))
+ FROM (J1_TBL CROSS JOIN J2_TBL)
+AS tx (ii, jj, tt, ii2, kk);
+
+-- [SPARK-28377] Fully support correlation names in the FROM clause
+-- SELECT '' AS `xxx`, tx.ii, tx.jj, tx.kk
+-- FROM (J1_TBL t1 (a, b, c) CROSS JOIN J2_TBL t2 (d, e))
+-- AS tx (ii, jj, tt, ii2, kk);
+
+SELECT udf('') AS `xxx`, udf(udf(j1_tbl.i)), udf(j), udf(t), udf(a.i),
udf(a.k), udf(b.i), udf(b.k)
+ FROM J1_TBL CROSS JOIN J2_TBL a CROSS JOIN J2_TBL b;
+
+
+--
+--
+-- Inner joins (equi-joins)
+--
+--
+
+--
+-- Inner joins (equi-joins) with USING clause
+-- The USING syntax changes the shape of the resulting table
+-- by including a column in the USING clause only once in the result.
+--
+
+-- Inner equi-join on specified column
+SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t), udf(k)
+ FROM J1_TBL INNER JOIN J2_TBL USING (i);
+
+-- Same as above, slightly different syntax
+SELECT udf(udf('')) AS `xxx`, udf(i), udf(j), udf(t), udf(k)
+ FROM J1_TBL JOIN J2_TBL USING (i);
+
+SELECT udf('') AS `xxx`, *
+ FROM J1_TBL t1 (a, b, c) JOIN J2_TBL t2 (a, d) USING (a)
+ ORDER BY udf(udf(a)), udf(d);
+
+-- [SPARK-28377] Fully support
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522321242 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522321244 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14362/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522321244 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14362/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522321242 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522321236 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14362/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522320231 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14362/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314984487
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -494,7 +506,27 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
case DateType =>
buildCast[Int](_, d => null)
case TimestampType =>
- buildCast[Long](_, t => timestampToLong(t).toShort)
+ buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (!failOnIntegerOverflow || longValue == longValue.toShort) {
+ longValue.toShort
+} else {
+ throw new ArithmeticException(s"Casting $t to Short causes
overflow.")
+}
+ })
+case x: NumericType if failOnIntegerOverflow =>
+ b =>
+val intValue = try {
+ x.exactNumeric.asInstanceOf[Numeric[Any]].toInt(b)
Review comment:
We cannot check the valid value range in a single place instead of the
current two checks in line 520 and 525?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314984213
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -494,7 +506,27 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
case DateType =>
buildCast[Int](_, d => null)
case TimestampType =>
- buildCast[Long](_, t => timestampToLong(t).toShort)
+ buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (!failOnIntegerOverflow || longValue == longValue.toShort) {
+ longValue.toShort
+} else {
+ throw new ArithmeticException(s"Casting $t to Short causes
overflow.")
+}
+ })
+case x: NumericType if failOnIntegerOverflow =>
+ b =>
+val intValue = try {
+ x.exactNumeric.asInstanceOf[Numeric[Any]].toInt(b)
Review comment:
Ah, I see.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314984060
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -1075,4 +1075,113 @@ class CastSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkEvaluation(cast("badvalue", dataType), null)
}
}
+
+ private def testIntMaxAndMin(dt: DataType): Unit = {
+Seq(Int.MaxValue + 1L, Int.MinValue - 1L).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, dt),
"overflow")
+
checkExceptionInExpression[ArithmeticException](cast(Decimal(value.toString),
dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast(Literal(value * MICROS_PER_SECOND, TimestampType), dt),
"overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast(Literal(value * 1.5f, FloatType), dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast(Literal(value * 1.0, DoubleType), dt), "overflow")
+}
+ }
+
+ private def testLongMaxAndMin(dt: DataType): Unit = {
+Seq(Decimal(Long.MaxValue) + Decimal(1), Decimal(Long.MinValue) -
Decimal(1)).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](
+cast(value, dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast((value * Decimal(1.1)).toFloat, dt), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+cast((value * Decimal(1.1)).toDouble, dt), "overflow")
+}
+ }
+
+ test("Cast to byte with option FAIL_ON_INTEGER_OVERFLOW enabled") {
+withSQLConf(SQLConf.FAIL_ON_INTEGER_OVERFLOW.key -> "true") {
+ testIntMaxAndMin(ByteType)
+ Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
+checkExceptionInExpression[ArithmeticException](cast(value, ByteType),
"overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType),
"overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
+ }
+
+ Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
+checkEvaluation(cast(value, ByteType), value)
+checkEvaluation(cast(value.toString, ByteType), value)
+checkEvaluation(cast(Decimal(value.toString), ByteType), value)
+checkEvaluation(cast(Literal(value * MICROS_PER_SECOND,
TimestampType), ByteType), value)
+checkEvaluation(cast(Literal(value.toInt, DateType), ByteType), null)
+checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType),
value)
+checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType),
value)
+ }
+}
+ }
+
+ test("Cast to short with option FAIL_ON_INTEGER_OVERFLOW enabled") {
+withSQLConf(SQLConf.FAIL_ON_INTEGER_OVERFLOW.key -> "true") {
+ testIntMaxAndMin(ShortType)
+ Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
+checkExceptionInExpression[ArithmeticException](cast(value,
ShortType), "overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType),
"overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
+checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
+ }
+
+ Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
+checkEvaluation(cast(value, ShortType), value)
+checkEvaluation(cast(value.toString, ShortType), value)
+checkEvaluation(cast(Decimal(value.toString), ShortType), value)
+checkEvaluation(cast(Literal(value * MICROS_PER_SECOND,
TimestampType), ShortType), value)
+checkEvaluation(cast(Literal(value.toInt, DateType), ShortType), null)
+checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType),
value)
+checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType),
value)
+ }
+}
+ }
+
+ test("Cast to int with option FAIL_ON_INTEGER_OVERFLOW enabled") {
+withSQLConf(SQLConf.FAIL_ON_INTEGER_OVERFLOW.key -> "true") {
+ testIntMaxAndMin(IntegerType)
+ testLongMaxAndMin(IntegerType)
+
+ Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
+checkEvaluation(cast(value, IntegerType), value)
+checkEvaluation(cast(value.toString, IntegerType), value)
+checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
+
[GitHub] [spark] SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
SparkQA commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522319180 **[Test build #109295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109295/testReport)** for PR 25423 at commit [`bae6524`](https://github.com/apache/spark/commit/bae6524d7d1965dbf30eb4146c1278e18af13801). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
HyukjinKwon edited a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522318935 @srowen and @shaneknapp, if the tests ^^^ fail, How about, for now, we go back to https://github.com/apache/spark/pull/25423/commits/f285b7094fbb6123f116e9ba7410408a43f8d256#diff-3c9f4fccf7d30ce2e8fa86db2ad1fdadR187 and just merge it? ```python # Test with Java 11 if "test-java11" in ghprb_pull_title: os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1" os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"], os.environ["PATH"]) ``` Because .. - this way is basically same as `hadoop-3.2` - so if anything should be fixed for this mechanism, we can fix it together in a separate PR if we should. - I think we couldn't conclude, yet: - if we want JDK 8 to build for both JDK 8 and 11, or .. - if we want JDK 11 (with `--release`) to build for both JDK 8 and 11. I am taking a look for [SPARK-28750](https://issues.apache.org/jira/browse/SPARK-28750) so maybe we can conclude the second point soon. Advantage of doing the suggested way is, we can at least build and run the tests with JDK 11 since the tests will pass now with #25443 without adding more changes in this PR. So, I was thinking: after this PR, we can conclude how we're going to handle JDK 8 or 11 to build, and make some more changes separately later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
HyukjinKwon edited a comment on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522318935 @srowen and @shaneknapp, if the tests ^^^ fail, How about, for now, we go back to https://github.com/apache/spark/pull/25423/commits/f285b7094fbb6123f116e9ba7410408a43f8d256#diff-3c9f4fccf7d30ce2e8fa86db2ad1fdadR187 and just merge it (both build and test with JDK 11)? ```python # Test with Java 11 if "test-java11" in ghprb_pull_title: os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1" os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"], os.environ["PATH"]) ``` Because .. - this way is basically same as `hadoop-3.2` - so if anything should be fixed for this mechanism, we can fix it together in a separate PR if we should. - I think we couldn't conclude, yet: - if we want JDK 8 to build for both JDK 8 and 11, or .. - if we want JDK 11 (with `--release`) to build for both JDK 8 and 11. I am taking a look for [SPARK-28750](https://issues.apache.org/jira/browse/SPARK-28750) so maybe we can conclude the second point soon. Advantage of doing the suggested way is, we can at least build and run the tests with JDK 11 since the tests will pass now with #25443 without adding more changes in this PR. So, I was thinking: after this PR, we can conclude how we're going to handle JDK 8 or 11 to build, and make some more changes separately later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
HyukjinKwon commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522318935 @srowen and @shaneknapp, if the tests ^^^ fail, How about, for now, we go back to https://github.com/apache/spark/pull/25423/commits/f285b7094fbb6123f116e9ba7410408a43f8d256#diff-3c9f4fccf7d30ce2e8fa86db2ad1fdadR187? ```python # Test with Java 11 if "test-java11" in ghprb_pull_title: os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1" os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"], os.environ["PATH"]) ``` Because .. - this way is basically same as `hadoop-3.2` - so if anything should be fixed for this mechanism, we can fix it together in a separate PR if we should. - I think we couldn't conclude, yet: - if we want JDK 8 to build for both JDK 8 and 11, or .. - if we want JDK 11 (with `--release`) to build for both JDK 8 and 11. I am taking a look for [SPARK-28750](https://issues.apache.org/jira/browse/SPARK-28750) so maybe we can conclude the second point soon. Advantage of doing the suggested way is, we can at least build and run the tests with JDK 11 since the tests will pass now with #25443 without adding more changes in this PR. So, I was thinking: after this PR, we can conclude how we're going to handle JDK 8 or 11 to build, and make some more changes separately later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds
HyukjinKwon commented on issue #25423: [SPARK-28701][test-hadoop3.2][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-522318785 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522317928 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522317930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14360/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522317920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522318080 **[Test build #109294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109294/testReport)** for PR 25443 at commit [`9defec2`](https://github.com/apache/spark/commit/9defec229199e690a5b25734ba3159182b382e3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522317922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14361/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522317928 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522317930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14360/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522317920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522317922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14361/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522317729 FYI, I send the confirmation that Hive 2.3.6 SNAPSHOT works fine with the current Spark master with JDK 11. Thanks @wangyum to trigger the tests against JDK 8 for sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522317427 **[Test build #109293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109293/testReport)** for PR 24829 at commit [`ad5363b`](https://github.com/apache/spark/commit/ad5363b0f02ff7bce73a7bbff31f171bb3bb2192). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
maropu commented on a change in pull request #25461: [SPARK-28741][SQL]Throw
exceptions when casting to integers causes overflow
URL: https://github.com/apache/spark/pull/25461#discussion_r314982581
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -1279,8 +1409,26 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
(c, evPrim, evNull) => code"$evNull = true;"
case TimestampType =>
(c, evPrim, evNull) => code"$evPrim = (long)
${timestampToIntegerCode(c)};"
+case DecimalType() if failOnIntegerOverflow =>
+ (c, evPrim, evNull) =>
+code"""
+ try {
+$evPrim = $c.toJavaBigInteger().longValueExact();
+ } catch (ArithmeticException e) {
+throw new ArithmeticException("Casting $c to Long causes
overflow");
+ }
+"""
case DecimalType() =>
(c, evPrim, evNull) => code"$evPrim = $c.toLong();"
+case _: FloatType | _: DoubleType if failOnIntegerOverflow =>
+ (c, evPrim, evNull) =>
+code"""
+ if ($c >= ${Long.MinValue}L && $c <= ${Long.MaxValue}L) {
Review comment:
Do we expect that this check matches the Nan case, too?
```
scala> sql("SET spark.sql.failOnIntegerOverFlow=true")
scala> sql("select cast(float('NaN') as long)").show
java.lang.ArithmeticException: Casting NaN to Long causes overflow.
at org.apache.spark.sql.types.FloatExactNumeric$.toLong(numerics.scala:137)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
