[GitHub] [spark] AmplabJenkins removed a comment on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double
AmplabJenkins removed a comment on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double URL: https://github.com/apache/spark/pull/24779#issuecomment-501245319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
SparkQA commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501246581 **[Test build #106421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106421/testReport)** for PR 24297 at commit [`4b6935b`](https://github.com/apache/spark/commit/4b6935bc5482d035a50607b1386a88aa7fcd805a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API
AmplabJenkins commented on issue #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API URL: https://github.com/apache/spark/pull/24851#issuecomment-501255901 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501208791 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501208798 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106407/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501225701 **[Test build #106416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106416/testReport)** for PR 24792 at commit [`3bf06af`](https://github.com/apache/spark/commit/3bf06affa4404cf884d2f3e41ea048ef19d23cff). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501225736 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501225747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106416/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501225736 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501219850 **[Test build #106416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106416/testReport)** for PR 24792 at commit [`3bf06af`](https://github.com/apache/spark/commit/3bf06affa4404cf884d2f3e41ea048ef19d23cff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
SparkQA commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501226092 **[Test build #106413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106413/testReport)** for PR 24850 at commit [`69470df`](https://github.com/apache/spark/commit/69470df00127fc9f1373d33fae90e8bd9d01dfa0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501228885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106411/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501228874 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
SparkQA commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501236551 **[Test build #106419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106419/testReport)** for PR 24297 at commit [`94b01fd`](https://github.com/apache/spark/commit/94b01fd9ab48abd2dec811328f62040adca24f7f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
AmplabJenkins removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501235925 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11663/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
AmplabJenkins removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501235920 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501235876 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501235885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11662/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501236553 **[Test build #106418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106418/testReport)** for PR 24792 at commit [`545bba0`](https://github.com/apache/spark/commit/545bba062649e3b23f0dcaa28592d017055be1cb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double
SparkQA commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double URL: https://github.com/apache/spark/pull/24779#issuecomment-501244770 **[Test build #106414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106414/testReport)** for PR 24779 at commit [`c36194e`](https://github.com/apache/spark/commit/c36194e3855fff8b51c786890a120dffd5ff4eec). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ketank-new commented on a change in pull request #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture
ketank-new commented on a change in pull request #24788: [SPARK-26985][CORE]
Fix "access only some column of the all of columns " for big endian architecture
URL: https://github.com/apache/spark/pull/24788#discussion_r292885145
##
File path: docs/sql-migration-guide-upgrade.md
##
@@ -23,6 +23,7 @@ license: |
{:toc}
## Upgrading From Spark SQL 2.4 to 3.0
+ - In Spark version 2.4 and earlier, SQL queries such as `FROM ` or
`FROM UNION ALL FROM ` are supported by accident. In hive-style
`FROM SELECT `, the `SELECT` clause is not negligible. Neither
Hive nor Presto support this syntax. Therefore we will treat these queries as
invalid since Spark 3.0.
Review comment:
yes .. looks like .. i better open a new PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501208798 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106407/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501208791 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501226300 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
AmplabJenkins commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501246979 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
SparkQA removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501246581 **[Test build #106421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106421/testReport)** for PR 24297 at commit [`4b6935b`](https://github.com/apache/spark/commit/4b6935bc5482d035a50607b1386a88aa7fcd805a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
AmplabJenkins commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501246983 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106421/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
SparkQA commented on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501246959 **[Test build #106421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106421/testReport)** for PR 24297 at commit [`4b6935b`](https://github.com/apache/spark/commit/4b6935bc5482d035a50607b1386a88aa7fcd805a). * This patch **fails Scala style tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `abstract class GraphElementFrame ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal
AmplabJenkins removed a comment on issue #24297: [SPARK-27299][GRAPH][WIP] Spark Graph API design proposal URL: https://github.com/apache/spark/pull/24297#issuecomment-501246979 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] s1ck commented on issue #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API
s1ck commented on issue #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API URL: https://github.com/apache/spark/pull/24851#issuecomment-501253442 @mengxr @gatorsmile Would be great to get some feedback from you. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API
AmplabJenkins commented on issue #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API URL: https://github.com/apache/spark/pull/24851#issuecomment-501253465 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501151660 **[Test build #106409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106409/testReport)** for PR 24829 at commit [`0a425c4`](https://github.com/apache/spark/commit/0a425c41b26225512cb9d0e8cb58986d76513f6c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501211351 **[Test build #106409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106409/testReport)** for PR 24829 at commit [`0a425c4`](https://github.com/apache/spark/commit/0a425c41b26225512cb9d0e8cb58986d76513f6c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501211943 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106409/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501211937 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501216857 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501216857 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501216863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11659/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501216863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11659/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture
SparkQA commented on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture URL: https://github.com/apache/spark/pull/24788#issuecomment-501219848 **[Test build #106417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106417/testReport)** for PR 24788 at commit [`9aa423f`](https://github.com/apache/spark/commit/9aa423f7e1e82cfc9b3046637ae35140ece6c960). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501219850 **[Test build #106416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106416/testReport)** for PR 24792 at commit [`3bf06af`](https://github.com/apache/spark/commit/3bf06affa4404cf884d2f3e41ea048ef19d23cff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
SparkQA commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501221884 **[Test build #106412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106412/testReport)** for PR 23850 at commit [`ba62794`](https://github.com/apache/spark/commit/ba62794bae0735762d0b4a2b4cd577c19383c6b3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture
AmplabJenkins removed a comment on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture URL: https://github.com/apache/spark/pull/24788#issuecomment-501221694 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11661/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture
AmplabJenkins removed a comment on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture URL: https://github.com/apache/spark/pull/24788#issuecomment-501221689 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
AmplabJenkins commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501222083 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
AmplabJenkins removed a comment on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501222083 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
SparkQA removed a comment on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501179373 **[Test build #106412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106412/testReport)** for PR 23850 at commit [`ba62794`](https://github.com/apache/spark/commit/ba62794bae0735762d0b4a2b4cd577c19383c6b3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
AmplabJenkins commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501222093 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106412/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
SparkQA removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501185342 **[Test build #106413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106413/testReport)** for PR 24850 at commit [`69470df`](https://github.com/apache/spark/commit/69470df00127fc9f1373d33fae90e8bd9d01dfa0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501226300 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501226304 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106413/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501225747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106416/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501228885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106411/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501228874 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double
AmplabJenkins commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double URL: https://github.com/apache/spark/pull/24779#issuecomment-501245319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double
AmplabJenkins commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double URL: https://github.com/apache/spark/pull/24779#issuecomment-501245330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106414/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double
SparkQA removed a comment on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double URL: https://github.com/apache/spark/pull/24779#issuecomment-501188256 **[Test build #106414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106414/testReport)** for PR 24779 at commit [`c36194e`](https://github.com/apache/spark/commit/c36194e3855fff8b51c786890a120dffd5ff4eec). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
SparkQA removed a comment on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501293697 **[Test build #106423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106423/testReport)** for PR 24852 at commit [`acfa50f`](https://github.com/apache/spark/commit/acfa50fdc61a08b7b4835b823c9223132282a190). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
AmplabJenkins removed a comment on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501329536 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24702: [SPARK-27989] [Kubernetes] [Core] Added retries on the connection to the driver for k8s
srowen commented on a change in pull request #24702: [SPARK-27989] [Kubernetes]
[Core] Added retries on the connection to the driver for k8s
URL: https://github.com/apache/spark/pull/24702#discussion_r292983218
##
File path:
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
##
@@ -283,7 +283,12 @@ private[spark] object CoarseGrainedExecutorBackend
extends Logging {
executorConf,
new SecurityManager(executorConf),
clientMode = true)
- val driver = fetcher.setupEndpointRefByURI(arguments.driverUrl)
+
+val driver = retry(3) {
+ fetcher.setupEndpointRefByURI(arguments.driverUrl)
+}
Review comment:
Oops @jlpedrosa I meant to retain the condition in the loop, but otherwise I
find the code right above simpler. However I like retaining the original
exception. Just please pay attention to the style. `driver = ` indent is off
and needs to say `if (i == nTries - 1)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24702: [SPARK-27989] [Kubernetes] [Core] Added retries on the connection to the driver for k8s
srowen commented on a change in pull request #24702: [SPARK-27989] [Kubernetes] [Core] Added retries on the connection to the driver for k8s URL: https://github.com/apache/spark/pull/24702#discussion_r292983407 ## File path: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile ## @@ -51,6 +51,8 @@ ENV SPARK_HOME /opt/spark WORKDIR /opt/spark/work-dir RUN chmod g+w /opt/spark/work-dir +#Disable negative dns reslolution https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html +RUN sed -i -e 's/networkaddress.cache.negative.ttl=10/networkaddress.cache.negative.ttl=0/g' /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/java.security Review comment: I just wouldn't make this change here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
AmplabJenkins removed a comment on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501329554 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106423/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
maryannxue commented on a change in pull request #24706: [SPARK-23128][SQL] A
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/24706#discussion_r292994737
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import java.util
+import java.util.concurrent.LinkedBlockingQueue
+
+import scala.collection.JavaConverters._
+import scala.collection.concurrent.TrieMap
+import scala.collection.mutable
+import scala.concurrent.ExecutionContext
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, ReturnAnswer}
+import org.apache.spark.sql.catalyst.rules.{Rule, RuleExecutor}
+import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec._
+import org.apache.spark.sql.execution.exchange._
+import
org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * A root node to execute the query plan adaptively. It splits the query plan
into independent
+ * stages and executes them in order according to their dependencies. The
query stage
+ * materializes its output at the end. When one stage completes, the data
statistics of the
+ * materialized output will be used to optimize the remainder of the query.
+ *
+ * To create query stages, we traverse the query tree bottom up. When we hit
an exchange node,
+ * and if all the child query stages of this exchange node are materialized,
we create a new
+ * query stage for this exchange node. The new stage is then materialized
asynchronously once it
+ * is created.
+ *
+ * When one query stage finishes materialization, the rest query is
re-optimized and planned based
+ * on the latest statistics provided by all materialized stages. Then we
traverse the query plan
+ * again and create more stages if possible. After all stages have been
materialized, we execute
+ * the rest of the plan.
+ */
+case class AdaptiveSparkPlanExec(
+initialPlan: SparkPlan,
+@transient session: SparkSession,
+@transient subqueryMap: Map[Long, ExecSubqueryExpression],
+@transient stageCache: TrieMap[SparkPlan, QueryStageExec])
+ extends LeafExecNode {
+
+ @transient private val executionId = Option(
+
session.sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)).map(_.toLong)
+
+ @transient private val lock = new Object()
+
+ // The logical plan optimizer for re-optimizing the current logical plan.
+ @transient private val optimizer = new RuleExecutor[LogicalPlan] {
+// TODO add more optimization rules
+override protected def batches: Seq[Batch] = Seq()
+ }
+
+ // A list of physical plan rules to be applied before creation of query
stages. The physical
+ // plan should reach a final status of query stages (i.e., no more addition
or removal of
+ // Exchange nodes) after running these rules.
+ @transient private val queryStagePreparationRules: Seq[Rule[SparkPlan]] =
Seq(
+PlanAdaptiveSubqueries(subqueryMap),
+EnsureRequirements(conf)
+ )
+
+ // A list of physical optimizer rules to be applied to a new stage before
its execution. These
+ // optimizations should be stage-independent.
+ @transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq(
+CollapseCodegenStages(conf)
+ )
+
+ private var currentStageId = 0
+
+ @volatile private var currentPhysicalPlan = initialPlan
+
+ @volatile private var isFinalPlan = false
+
+ @volatile private var fallback = false
+
+ /**
+ * Return type for `createQueryStages`
+ * @param newPlan the new plan with created query stages.
+ * @param allChildStagesMaterialized whether all child stages have been
materialized.
+ * @param newStages the newly created query stages, including new reused
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24851: [SPARK-27303][GRAPH] Add PropertyGraph construction API
dongjoon-hyun commented on a change in pull request #24851:
[SPARK-27303][GRAPH] Add PropertyGraph construction API
URL: https://github.com/apache/spark/pull/24851#discussion_r292997721
##
File path:
graph/api/src/test/java/org/apache/spark/graph/api/JavaPropertyGraphSuite.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.graph.api;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.test.TestSparkSession;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+public abstract class JavaPropertyGraphSuite implements Serializable {
+private transient TestSparkSession spark;
+private transient CypherSession cypherSession = null;
+
+@Before
+public void setUp() {
+spark = new TestSparkSession();
+}
+
+@After
+public void tearDown() {
+spark.stop();
+spark = null;
+}
+
+@Test
+public void testCreateFromNodeFrame() {
+StructType personSchema = createSchema(
+Lists.newArrayList("id", "name"),
+Lists.newArrayList(DataTypes.LongType, DataTypes.StringType));
+
+List personData = Arrays.asList(
+RowFactory.create(0L, "Alice"),
+RowFactory.create(1L, "Bob"));
+
+StructType knowsSchema = createSchema(
+Lists.newArrayList("id", "source", "target", "since"),
+Lists.newArrayList(DataTypes.LongType, DataTypes.LongType,
DataTypes.LongType, DataTypes.IntegerType));
+
+List knowsData = Collections.singletonList(RowFactory.create(0L,
0L, 1L, 1984));
+
+Dataset personDf = spark.createDataFrame(personData,
personSchema);
+NodeFrame personNodeFrame = NodeFrame.create(personDf, "id",
Sets.newHashSet("Person"));
+
+Dataset knowsDf = spark.createDataFrame(knowsData, knowsSchema);
+RelationshipFrame knowsRelFrame = RelationshipFrame.create(knowsDf,
"id", "source", "target", "KNOWS");
+
+PropertyGraph graph =
cypherSession.createGraph(Lists.newArrayList(personNodeFrame),
Lists.newArrayList(knowsRelFrame));
+List result = graph.nodes().collectAsList();
+Assert.assertEquals(1, result.size());
+}
+
+private StructType createSchema(List fieldNames, List
dataTypes) {
+List fields = new ArrayList<>();
+for (int i = 0; i < fieldNames.size(); i++) {
+fields.add(DataTypes.createStructField(fieldNames.get(i),
dataTypes.get(i), true));
+}
+return DataTypes.createStructType(fields);
+}
+}
Review comment:
FYI, Apache Spark uses 2-space indentation for Java code, too. Please fix
this one and consider that in the next PRs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24855: [SPARK-28030] convert filePath to URI in binary file data source
SparkQA commented on issue #24855: [SPARK-28030] convert filePath to URI in binary file data source URL: https://github.com/apache/spark/pull/24855#issuecomment-501345556 **[Test build #106428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106428/testReport)** for PR 24855 at commit [`cf96785`](https://github.com/apache/spark/commit/cf96785ca77839ff92b7d9be1ccb67fb009be04b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column
SparkQA commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column URL: https://github.com/apache/spark/pull/24854#issuecomment-501345557 **[Test build #106429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106429/testReport)** for PR 24854 at commit [`6afd936`](https://github.com/apache/spark/commit/6afd9362b597576e40860327a771fadcc9269078). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
SparkQA commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501345570 **[Test build #106430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106430/testReport)** for PR 24706 at commit [`ec59f88`](https://github.com/apache/spark/commit/ec59f882cf0b75fba7a4e0c0521db2ab9075f9f2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] arunmahadevan commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode
arunmahadevan commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-501348243
Output watermark can be computed as some function of input watermark and the
timestamp of events at that operator (e.g. min(input watermarks, timestamps of
oldest event at that node)) so we could compute the other by storing only the
input watermark. For now, we require the user to provide a timestamp column +
lag using “withWatermark()” before each aggregate operation.
E.g.
```
input.
.withWatermark("inputtime",...)
.groupBy(window(...))
.select($"window1.end".as("windowtime")
.withWatermark("windowtime")..
.groupBy(...)...
```
Here the window.end of the first groupBy is the output watermark which
becomes the input watermark of the second groupBy.
Also note that the input water mark of an operator is propagated to the next
operator only in the next batch so that it processes the events first and then
the watermark.
Let me know the specific cases where you found issues.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #24855: [SPARK-28030] convert filePath to URI in binary file data source
holdenk commented on a change in pull request #24855: [SPARK-28030] convert
filePath to URI in binary file data source
URL: https://github.com/apache/spark/pull/24855#discussion_r293003817
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala
##
@@ -368,4 +368,14 @@ class BinaryFileFormatSuite extends QueryTest with
SharedSQLContext with SQLTest
assert(caught.getMessage.contains("exceeds the max length allowed"))
}
}
+
+ test("SPARK-28030: support chars in file names that require URL encoding") {
+withTempDir { dir =>
+ val file = new File(dir, "test space.txt")
+ val content = "123".getBytes
+ Files.write(file.toPath, content, StandardOpenOption.CREATE,
StandardOpenOption.WRITE)
+ val df = spark.read.format(BINARY_FILE).load(dir.getPath)
+ df.collect()
Review comment:
Maybe it would be good to assert that we get the bytes back, just in case
it's skipping the files with space or something.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #24855: [SPARK-28030] convert filePath to URI in binary file data source
holdenk commented on a change in pull request #24855: [SPARK-28030] convert
filePath to URI in binary file data source
URL: https://github.com/apache/spark/pull/24855#discussion_r293004219
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala
##
@@ -368,4 +368,14 @@ class BinaryFileFormatSuite extends QueryTest with
SharedSQLContext with SQLTest
assert(caught.getMessage.contains("exceeds the max length allowed"))
}
}
+
+ test("SPARK-28030: support chars in file names that require URL encoding") {
+withTempDir { dir =>
Review comment:
Is it ok that we only have the space in the file name, or do we need it in
the path were providing to trigger SPARK-28030?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #24855: [SPARK-28030] convert filePath to URI in binary file data source
holdenk commented on a change in pull request #24855: [SPARK-28030] convert filePath to URI in binary file data source URL: https://github.com/apache/spark/pull/24855#discussion_r293006022 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala ## @@ -38,7 +38,7 @@ import org.apache.spark.util.NextIterator * that need to be prepended to each row. * * @param partitionValues value of partition columns to be prepended to each row. - * @param filePath path of the file to read Review comment: nit: Maybe add a comment here that we throw an exception if were passed an invalid URI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #24855: [SPARK-28030] convert filePath to URI in binary file data source
holdenk commented on a change in pull request #24855: [SPARK-28030] convert
filePath to URI in binary file data source
URL: https://github.com/apache/spark/pull/24855#discussion_r293004966
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala
##
@@ -368,4 +368,14 @@ class BinaryFileFormatSuite extends QueryTest with
SharedSQLContext with SQLTest
assert(caught.getMessage.contains("exceeds the max length allowed"))
}
}
+
+ test("SPARK-28030: support chars in file names that require URL encoding") {
Review comment:
The change seems to impact not just the binary file format, maybe this
belongs in one of our root datasource tests. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WangGuangxin commented on a change in pull request #24043: [SPARK-11412][SQL] Support merge schema for ORC
WangGuangxin commented on a change in pull request #24043: [SPARK-11412][SQL]
Support merge schema for ORC
URL: https://github.com/apache/spark/pull/24043#discussion_r293006575
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ReadSchemaSuite.scala
##
@@ -134,6 +135,31 @@ class VectorizedOrcReadSchemaSuite
}
}
+class MergedOrcReadSchemaSuite
+ extends ReadSchemaSuite
+ with AddColumnIntoTheMiddleTest
+ with HideColumnInTheMiddleTest
+ with AddNestedColumnTest
+ with HideNestedColumnTest
+ with ChangePositionTest
+ with BooleanTypeTest
+ with IntegralTypeTest
+ with ToDoubleTypeTest {
+
+ override val format: String = "orc"
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column
AmplabJenkins commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column URL: https://github.com/apache/spark/pull/24854#issuecomment-501357776 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106429/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column
AmplabJenkins commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column URL: https://github.com/apache/spark/pull/24854#issuecomment-501357769 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on issue #24833: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`
zsxwing commented on issue #24833: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py` URL: https://github.com/apache/spark/pull/24833#issuecomment-501363039 @srowen @dongjoon-hyun could you also fix the jenkins job since it's broken by this PR? https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-maven-hadoop-2.7/ ``` Unable to parse command line options: Unrecognized option: --force ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24857: [CORE] Remove an unused variable in SparkSubmt.scala
AmplabJenkins removed a comment on issue #24857: [CORE] Remove an unused variable in SparkSubmt.scala URL: https://github.com/apache/spark/pull/24857#issuecomment-501364955 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24857: [CORE] Remove an unused variable in SparkSubmt.scala
AmplabJenkins commented on issue #24857: [CORE] Remove an unused variable in SparkSubmt.scala URL: https://github.com/apache/spark/pull/24857#issuecomment-501366199 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24842: [SPARK-28002][SQL] Support WITH clause column aliases
dongjoon-hyun commented on a change in pull request #24842: [SPARK-28002][SQL]
Support WITH clause column aliases
URL: https://github.com/apache/spark/pull/24842#discussion_r293025899
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
##
@@ -633,4 +634,16 @@ class AnalysisSuite extends AnalysisTest with Matchers {
val res = ViewAnalyzer.execute(view)
comparePlans(res, expected)
}
+
+ test("SPARK-28002: CTE with non-existing column alias") {
+assertAnalysisError(parsePlan("WITH t(x) AS (SELECT 1) SELECT * FROM t
WHERE y = 1"),
+ Seq("cannot resolve '`y`' given input columns: [x]"))
+ }
+
+ test("SPARK-28002: CTE with non-matching column alias") {
+assertAnalysisError(parsePlan("WITH t(x, y) AS (SELECT 1) SELECT * FROM t
WHERE x = 1"),
+ Seq("Number of column aliases does not match number of columns. Number
of column aliases: " +
+"2; number of columns: 1."))
+ }
+
Review comment:
nit. remove empty line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24833: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`
srowen commented on issue #24833: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py` URL: https://github.com/apache/spark/pull/24833#issuecomment-501374904 Oh, nevermind. The _compile_ job failed and I hadn't updated that. I had updates the _test_ jobs previously. I don't think anything was overwritten. OK, should be OK now unless I'm missing more master jobs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] BryanCutler commented on a change in pull request #24844: [SPARK-28003][PYTHON] Allow NaT values when creating Spark dataframe from pandas with Arrow
BryanCutler commented on a change in pull request #24844: [SPARK-28003][PYTHON] Allow NaT values when creating Spark dataframe from pandas with Arrow URL: https://github.com/apache/spark/pull/24844#discussion_r293034013 ## File path: python/pyspark/serializers.py ## @@ -296,7 +296,7 @@ def create_array(s, t): mask = s.isnull() # Ensure timestamp series are in expected form for Spark internal representation if t is not None and pa.types.is_timestamp(t): -s = _check_series_convert_timestamps_internal(s.fillna(0), self._timezone) +s = _check_series_convert_timestamps_internal(s, self._timezone) Review comment: I believe it was due to a Pandas error, most likely because we were testing with 0.19.2 at the time. Can you manually run some tests with different Pandas versions? It will be best to test with older versions, but it might be kind of hard to get 0.19.2 working with pyarrow 0.12.1 though.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] BryanCutler commented on a change in pull request #24844: [SPARK-28003][PYTHON] Allow NaT values when creating Spark dataframe from pandas with Arrow
BryanCutler commented on a change in pull request #24844: [SPARK-28003][PYTHON]
Allow NaT values when creating Spark dataframe from pandas with Arrow
URL: https://github.com/apache/spark/pull/24844#discussion_r293038513
##
File path: python/pyspark/sql/tests/test_arrow.py
##
@@ -383,6 +383,19 @@ def test_timestamp_dst(self):
assert_frame_equal(pdf, df_from_python.toPandas())
assert_frame_equal(pdf, df_from_pandas.toPandas())
+def test_timestamp_nat(self):
+import pandas as pd
+dt1 = [pd.NaT, pd.Timestamp('2019-06-11')] * 100
+dt2 = [None, pd.Timestamp('2019-06-11')] * 100
+pdf1 = pd.DataFrame({'time': dt1})
+pdf2 = pd.DataFrame({'time': dt2})
+
+df1 = self.spark.createDataFrame(pdf1)
+df2 = self.spark.createDataFrame(pdf2)
Review comment:
I think you can just combine these to 1 DataFrame, but I think it would be
good to also check against toPandas without Arrow
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
gatorsmile commented on a change in pull request #24706: [SPARK-23128][SQL] A
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/24706#discussion_r292898739
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.{ReusedSubqueryExec, SparkPlan}
+import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec,
SortMergeJoinExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSQLContext
+
+class AdaptiveQueryExecSuite extends QueryTest with SharedSQLContext {
+
+ setupTestData()
+
+ private def runAdaptiveAndVerifyResult(query: String): (SparkPlan,
SparkPlan) = {
+val dfAdaptive = sql(query)
+val result = dfAdaptive.collect()
+withSQLConf(SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "false") {
+ val df = sql(query)
+ QueryTest.sameRows(result.toSeq, df.collect().toSeq)
+}
+val plan = dfAdaptive.queryExecution.executedPlan
+assert(plan.isInstanceOf[AdaptiveSparkPlanExec])
+val adaptivePlan = plan.asInstanceOf[AdaptiveSparkPlanExec].executedPlan
+(dfAdaptive.queryExecution.sparkPlan, adaptivePlan)
+ }
+
+ private def findTopLevelBroadcastHashJoin(plan: SparkPlan):
Seq[BroadcastHashJoinExec] = {
+plan.collect {
+ case j: BroadcastHashJoinExec => Seq(j)
+ case s: QueryStageExec => findTopLevelBroadcastHashJoin(s.plan)
+}.flatten
+ }
+
+ private def findTopLevelSortMergeJoin(plan: SparkPlan):
Seq[SortMergeJoinExec] = {
+plan.collect {
+ case j: SortMergeJoinExec => Seq(j)
+ case s: QueryStageExec => findTopLevelSortMergeJoin(s.plan)
+}.flatten
+ }
+
+ private def findReusedExchange(plan: SparkPlan): Seq[ReusedQueryStageExec] =
{
+plan.collect {
+ case e: ReusedQueryStageExec => Seq(e)
+ case a: AdaptiveSparkPlanExec => findReusedExchange(a.executedPlan)
+ case s: QueryStageExec => findReusedExchange(s.plan)
+ case p: SparkPlan => p.subqueries.flatMap(findReusedExchange)
+}.flatten
+ }
+
+ private def findReusedSubquery(plan: SparkPlan): Seq[ReusedSubqueryExec] = {
+plan.collect {
+ case e: ReusedSubqueryExec => Seq(e)
+ case s: QueryStageExec => findReusedSubquery(s.plan)
+ case p: SparkPlan => p.subqueries.flatMap(findReusedSubquery)
+}.flatten
+ }
+
+ test("Change merge join to broadcast join") {
+withSQLConf(
+SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "true",
+SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "80") {
+ val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(
+"SELECT * FROM testData join testData2 ON key = a where value = '1'")
Review comment:
```Scala
withSQLConf(
SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "true",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "80") {
spark.sql(
"SELECT * FROM testData join testData2 ON key = a where value =
'1'").explain()
}
withSQLConf(
SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "false",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "80") {
spark.sql(
"SELECT * FROM testData join testData2 ON key = a where value =
'1'").explain()
}
```
Try these? It sounds like we are unable to tell the difference from the
physical plans. Also, the WholeStage codegen ID is lost after we turn on the
adaptive query execution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
[GitHub] [spark] gatorsmile commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
gatorsmile commented on a change in pull request #24706: [SPARK-23128][SQL] A
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/24706#discussion_r292898739
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.execution.{ReusedSubqueryExec, SparkPlan}
+import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec,
SortMergeJoinExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSQLContext
+
+class AdaptiveQueryExecSuite extends QueryTest with SharedSQLContext {
+
+ setupTestData()
+
+ private def runAdaptiveAndVerifyResult(query: String): (SparkPlan,
SparkPlan) = {
+val dfAdaptive = sql(query)
+val result = dfAdaptive.collect()
+withSQLConf(SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "false") {
+ val df = sql(query)
+ QueryTest.sameRows(result.toSeq, df.collect().toSeq)
+}
+val plan = dfAdaptive.queryExecution.executedPlan
+assert(plan.isInstanceOf[AdaptiveSparkPlanExec])
+val adaptivePlan = plan.asInstanceOf[AdaptiveSparkPlanExec].executedPlan
+(dfAdaptive.queryExecution.sparkPlan, adaptivePlan)
+ }
+
+ private def findTopLevelBroadcastHashJoin(plan: SparkPlan):
Seq[BroadcastHashJoinExec] = {
+plan.collect {
+ case j: BroadcastHashJoinExec => Seq(j)
+ case s: QueryStageExec => findTopLevelBroadcastHashJoin(s.plan)
+}.flatten
+ }
+
+ private def findTopLevelSortMergeJoin(plan: SparkPlan):
Seq[SortMergeJoinExec] = {
+plan.collect {
+ case j: SortMergeJoinExec => Seq(j)
+ case s: QueryStageExec => findTopLevelSortMergeJoin(s.plan)
+}.flatten
+ }
+
+ private def findReusedExchange(plan: SparkPlan): Seq[ReusedQueryStageExec] =
{
+plan.collect {
+ case e: ReusedQueryStageExec => Seq(e)
+ case a: AdaptiveSparkPlanExec => findReusedExchange(a.executedPlan)
+ case s: QueryStageExec => findReusedExchange(s.plan)
+ case p: SparkPlan => p.subqueries.flatMap(findReusedExchange)
+}.flatten
+ }
+
+ private def findReusedSubquery(plan: SparkPlan): Seq[ReusedSubqueryExec] = {
+plan.collect {
+ case e: ReusedSubqueryExec => Seq(e)
+ case s: QueryStageExec => findReusedSubquery(s.plan)
+ case p: SparkPlan => p.subqueries.flatMap(findReusedSubquery)
+}.flatten
+ }
+
+ test("Change merge join to broadcast join") {
+withSQLConf(
+SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "true",
+SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "80") {
+ val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(
+"SELECT * FROM testData join testData2 ON key = a where value = '1'")
Review comment:
```Scala
withSQLConf(
SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "true",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "80") {
spark.sql(
"SELECT * FROM testData join testData2 ON key = a where value =
'1'").explain()
}
withSQLConf(
SQLConf.RUNTIME_REOPTIMIZATION_ENABLED.key -> "false",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "80") {
spark.sql(
"SELECT * FROM testData join testData2 ON key = a where value =
'1'").explain()
}
```
Try these? It sounds like the WholeStage codegen ID is lost after we turn on
the adaptive query execution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
dongjoon-hyun closed pull request #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501307990 **[Test build #106418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106418/testReport)** for PR 24792 at commit [`545bba0`](https://github.com/apache/spark/commit/545bba062649e3b23f0dcaa28592d017055be1cb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501318777 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501318798 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11667/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql
SparkQA commented on issue #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql URL: https://github.com/apache/spark/pull/24853#issuecomment-501323266 **[Test build #106425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106425/testReport)** for PR 24853 at commit [`6cf1d6a`](https://github.com/apache/spark/commit/6cf1d6a6fe52e909ca311e83a1c36fe3e1f831f3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql
AmplabJenkins commented on issue #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql URL: https://github.com/apache/spark/pull/24853#issuecomment-501322419 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql
AmplabJenkins commented on issue #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql URL: https://github.com/apache/spark/pull/24853#issuecomment-501322435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11668/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24853: [WIP][SPARK-28029][SQL][TEST] Port int2.sql
wangyum commented on a change in pull request #24853:
[WIP][SPARK-28029][SQL][TEST] Port int2.sql
URL: https://github.com/apache/spark/pull/24853#discussion_r292972799
##
File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/int2.sql
##
@@ -0,0 +1,126 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- INT2
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/int2.sql
+
+CREATE TABLE INT2_TBL(f1 smallint) USING parquet;
+
+-- [SPARK-28023] Trim the string when cast string type to other types
+INSERT INTO INT2_TBL VALUES (trim('0 '));
+
+INSERT INTO INT2_TBL VALUES (trim(' 1234 '));
+
+INSERT INTO INT2_TBL VALUES (trim('-1234'));
+
+-- [SPARK-27923] Invalid input syntax for type short throws exception at
PostgreSQL
+-- INSERT INTO INT2_TBL VALUES ('34.5');
+
+-- largest and smallest values
+INSERT INTO INT2_TBL VALUES ('32767');
+
+INSERT INTO INT2_TBL VALUES ('-32767');
+
+-- bad input values -- should give errors
+-- INSERT INTO INT2_TBL VALUES ('10');
+-- INSERT INTO INT2_TBL VALUES ('asdf');
+-- INSERT INTO INT2_TBL VALUES ('');
+-- INSERT INTO INT2_TBL VALUES ('- 1234');
+-- INSERT INTO INT2_TBL VALUES ('4 444');
+-- INSERT INTO INT2_TBL VALUES ('123 dt');
+-- INSERT INTO INT2_TBL VALUES ('');
+
+
+SELECT '' AS five, * FROM INT2_TBL;
+
+SELECT '' AS four, i.* FROM INT2_TBL i WHERE i.f1 <> smallint('0');
+
+SELECT '' AS four, i.* FROM INT2_TBL i WHERE i.f1 <> int('0');
+
+SELECT '' AS one, i.* FROM INT2_TBL i WHERE i.f1 = smallint('0');
+
+SELECT '' AS one, i.* FROM INT2_TBL i WHERE i.f1 = int('0');
+
+SELECT '' AS two, i.* FROM INT2_TBL i WHERE i.f1 < smallint('0');
+
+SELECT '' AS two, i.* FROM INT2_TBL i WHERE i.f1 < int('0');
+
+SELECT '' AS three, i.* FROM INT2_TBL i WHERE i.f1 <= smallint('0');
+
+SELECT '' AS three, i.* FROM INT2_TBL i WHERE i.f1 <= int('0');
+
+SELECT '' AS two, i.* FROM INT2_TBL i WHERE i.f1 > smallint('0');
+
+SELECT '' AS two, i.* FROM INT2_TBL i WHERE i.f1 > int('0');
+
+SELECT '' AS three, i.* FROM INT2_TBL i WHERE i.f1 >= smallint('0');
+
+SELECT '' AS three, i.* FROM INT2_TBL i WHERE i.f1 >= int('0');
+
+-- positive odds
+SELECT '' AS one, i.* FROM INT2_TBL i WHERE (i.f1 % smallint('2')) =
smallint('1');
+
+-- any evens
+SELECT '' AS three, i.* FROM INT2_TBL i WHERE (i.f1 % int('2')) =
smallint('0');
+
+-- [SPARK-28024] Incorrect value when out of range
+SELECT '' AS five, i.f1, i.f1 * smallint('2') AS x FROM INT2_TBL i;
+
+SELECT '' AS five, i.f1, i.f1 * smallint('2') AS x FROM INT2_TBL i
+WHERE abs(f1) < 16384;
+
+SELECT '' AS five, i.f1, i.f1 * int('2') AS x FROM INT2_TBL i;
+
+-- [SPARK-28024] Incorrect value when out of range
+SELECT '' AS five, i.f1, i.f1 + smallint('2') AS x FROM INT2_TBL i;
+
+SELECT '' AS five, i.f1, i.f1 + smallint('2') AS x FROM INT2_TBL i
+WHERE f1 < 32766;
+
+SELECT '' AS five, i.f1, i.f1 + int('2') AS x FROM INT2_TBL i;
+
+-- [SPARK-28024] Incorrect value when out of range
+SELECT '' AS five, i.f1, i.f1 - smallint('2') AS x FROM INT2_TBL i;
+
+SELECT '' AS five, i.f1, i.f1 - smallint('2') AS x FROM INT2_TBL i
+WHERE f1 > -32767;
+
+SELECT '' AS five, i.f1, i.f1 - int('2') AS x FROM INT2_TBL i;
+
+-- The result is different because [SPARK-2659]
+SELECT '' AS five, i.f1, i.f1 / smallint('2') AS x FROM INT2_TBL i;
+
+-- The result is different because [SPARK-2659]
+SELECT '' AS five, i.f1, i.f1 / int('2') AS x FROM INT2_TBL i;
+
+-- corner cases
+SELECT string(shiftleft(smallint(-1), 15));
+SELECT string(smallint(shiftleft(smallint(-1), 15))+1);
+
+-- check sane handling of INT16_MIN overflow cases
+-- [SPARK-28024] Incorrect numeric values when out of range
+-- SELECT smallint((-32768)) * smallint(-1);
+-- SELECT smallint(-32768) / smallint(-1);
+SELECT smallint(-32768) % smallint(-1);
+
+-- [SPARK-28028] Cast numeric to integral type need round
+-- check rounding when casting from float
+SELECT x, smallint(x) AS int2_value
+FROM (VALUES float(-2.5),
+ float(-1.5),
+ float(-0.5),
+ float(0.0),
+ float(0.5),
+ float(1.5),
+ float(2.5)) t(x);
+
+-- check rounding when casting from numeric
+SELECT x, smallint(x) AS int2_value
+FROM (VALUES decimal(-2.5),
Review comment:
We do not have `numeric` type, can we use `decimal`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
SparkQA removed a comment on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501254287 **[Test build #106422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106422/testReport)** for PR 23850 at commit [`f8adab3`](https://github.com/apache/spark/commit/f8adab32b0eec1ae96f124a563f4393bf3db6659). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
SparkQA commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501329288 **[Test build #106423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106423/testReport)** for PR 24852 at commit [`acfa50f`](https://github.com/apache/spark/commit/acfa50fdc61a08b7b4835b823c9223132282a190). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
AmplabJenkins commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501329536 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
AmplabJenkins commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501329554 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106423/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
SparkQA commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501329684 **[Test build #106422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106422/testReport)** for PR 23850 at commit [`f8adab3`](https://github.com/apache/spark/commit/f8adab32b0eec1ae96f124a563f4393bf3db6659). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum closed pull request #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
wangyum closed pull request #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators
wangyum commented on issue #24852: [SPARK-28027][SQL] Add bitwise shift left/right operators URL: https://github.com/apache/spark/pull/24852#issuecomment-501335275 Close it because it conflicts with the table definition: ``` CREATE TABLE parquet_compat( col_0 ARRAY, col_1 ARRAY, col_2 ARRAY> ---^^^ ) ``` https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106423/testReport/org.apache.spark.sql.hive/ParquetHiveCompatibilitySuite/array/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on issue #24643: [SPARK-26412][PySpark][SQL] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
WeichenXu123 commented on issue #24643: [SPARK-26412][PySpark][SQL] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-501337363 @meng Done. Leave doc which will update soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
