[GitHub] [spark] SparkQA commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
SparkQA commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529313 **[Test build #107954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107954/testReport)** for PR 25213 at commit [`0ae5dad`](https://github.com/apache/spark/commit/0ae5dadafed72d65566585c059c7f767db36bb2f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
SparkQA commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529316 **[Test build #107956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107956/testReport)** for PR 25214 at commit [`832d009`](https://github.com/apache/spark/commit/832d009a69a09cde7011712f32a5afb460c8ef7c). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-513529312 **[Test build #107957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107957/testReport)** for PR 22282 at commit [`0a73fdc`](https://github.com/apache/spark/commit/0a73fdc1a3faf0484d98e26259e2c2b9e195f53d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing
SparkQA commented on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513529311 **[Test build #107953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107953/testReport)** for PR 25184 at commit [`2c354a0`](https://github.com/apache/spark/commit/2c354a09d6baa9dbbda10050d75716adc7493472). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
SparkQA commented on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216#issuecomment-513529314 **[Test build #107952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107952/testReport)** for PR 25216 at commit [`a036161`](https://github.com/apache/spark/commit/a036161edf2fa70ded367a9ac0a15d0213b32619). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529354 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107956/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-513529348 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107957/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-513529345 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529353 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
SparkQA removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-513524665 **[Test build #107957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107957/testReport)** for PR 22282 at commit [`0a73fdc`](https://github.com/apache/spark/commit/0a73fdc1a3faf0484d98e26259e2c2b9e195f53d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
SparkQA removed a comment on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216#issuecomment-513515832 **[Test build #107952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107952/testReport)** for PR 25216 at commit [`a036161`](https://github.com/apache/spark/commit/a036161edf2fa70ded367a9ac0a15d0213b32619). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing
SparkQA removed a comment on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513516776 **[Test build #107953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107953/testReport)** for PR 25184 at commit [`2c354a0`](https://github.com/apache/spark/commit/2c354a09d6baa9dbbda10050d75716adc7493472). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-513529345 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529364 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107954/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529362 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529353 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
SparkQA removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513521210 **[Test build #107956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107956/testReport)** for PR 25214 at commit [`832d009`](https://github.com/apache/spark/commit/832d009a69a09cde7011712f32a5afb460c8ef7c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing
AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513529381 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
SparkQA removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513518637 **[Test build #107954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107954/testReport)** for PR 25213 at commit [`0ae5dad`](https://github.com/apache/spark/commit/0ae5dadafed72d65566585c059c7f767db36bb2f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
AmplabJenkins commented on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216#issuecomment-513529414 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
AmplabJenkins commented on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216#issuecomment-513529416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107952/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529362 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing
AmplabJenkins commented on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513529383 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107953/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-513529348 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107957/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
AmplabJenkins removed a comment on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216#issuecomment-513529414 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529354 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107956/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing
AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513529381 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529364 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107954/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing
AmplabJenkins removed a comment on issue #25184: [SPARK-28431][SQL] Set meximum error message length in CSV datasource's parsing and writing URL: https://github.com/apache/spark/pull/25184#issuecomment-513529383 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107953/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
AmplabJenkins removed a comment on issue #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216#issuecomment-513529416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107952/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
dongjoon-hyun commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529518 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
wangyum commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529566 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun edited a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513529644 All test passed and only CRAN check fails. The failure is irrelevant to this PR. ``` * checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : dims [product 24] do not match the length of object [0] Execution halted ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513529644 All test passed and only CRAN check fails. ``` * checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : dims [product 24] do not match the length of object [0] Execution halted ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite
dongjoon-hyun closed pull request #25216: [SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite URL: https://github.com/apache/spark/pull/25216 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529827 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13067/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529844 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529823 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529845 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529844 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529823 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529827 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13067/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
AmplabJenkins removed a comment on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529845 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures
SparkQA commented on issue #25213: [SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures URL: https://github.com/apache/spark/pull/25213#issuecomment-513529974 **[Test build #107959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107959/testReport)** for PR 25213 at commit [`0ae5dad`](https://github.com/apache/spark/commit/0ae5dadafed72d65566585c059c7f767db36bb2f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
SparkQA commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-513529973 **[Test build #107958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107958/testReport)** for PR 25214 at commit [`832d009`](https://github.com/apache/spark/commit/832d009a69a09cde7011712f32a5afb460c8ef7c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25215: [SPARK-28445][SQL][Python] Fix error when PythonUDF is used in both group by and aggregate expression
viirya commented on issue #25215: [SPARK-28445][SQL][Python] Fix error when PythonUDF is used in both group by and aggregate expression URL: https://github.com/apache/spark/pull/25215#issuecomment-513530717 ``` * checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : dims [product 24] do not match the length of object [0] ``` SparkR CRAN feasibility check (SPARK-24152) fails again..Emailed to CRAN for help. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305604790 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1605,3 +1605,49 @@ private case class GetTimestamp( override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression = copy(timeZoneId = Option(timeZoneId)) } + +@ExpressionDescription( + usage = "_FUNC_(year, month, day) - Create date from year, month and day fields.", + arguments = """ +Arguments: + * year - the year to represent, from 1 to + * month - the month-of-year to represent, from 1 (January) to 12 (December) + * day - the day-of-month to represent, from 1 to 31 + """, + examples = """ +Examples: + > SELECT _FUNC_(2013, 7, 15); + '2013-07-15' Review comment: We don't have the single quotation marks around the date. This example should be the result of `spark-sql`. Where did you get the above result? ``` spark-sql> SELECT make_date(2013, 7, 15); 2013-07-15 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305604870 ## File path: sql/core/src/test/resources/sql-tests/inputs/datetime.sql ## @@ -34,3 +34,5 @@ select date '2001-09-28' + 7; select 7 + date '2001-09-28'; select date '2001-10-01' - 7; select date '2001-10-01' - date '2001-09-28'; + +select make_date(2013, 7, 15); Review comment: Please revert these `datetime.sql` and `datetime.sql.out`. Instead, please update `date.sql`. - sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql You need to uncomment the followings. ``` -- select make_date(2013, 7, 15); -- select make_date(-44, 3, 15); -- select make_time(8, 20, 0.0); -- should fail -- select make_date(2013, 2, 30); -- select make_date(2013, 13, 1); -- select make_date(2013, 11, -1); -- select make_time(10, 55, 100.1); -- select make_time(24, 0, 2.1); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-513522547 And parser should recognises the DDL type string ideally (needs to check Hive) and R ser/de should recognise it too ideally. We should provide a way to define interval instances in Python and R sides too ideally. Otherwise, we have no way to use it in Python and R native APIs. For instance, time delta in Python should be matched to interval. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-513522547 And parser should recognises the DDL type string ideally (needs to check Hive) and R ser/de should recognises it too ideally. We should provide a way to define interval instances in Python and R sides too ideally. Otherwise, we have no way to use it in Python and R native APIs. For instance, time delta in Python should be matched to interval. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
cloud-fan commented on a change in pull request #24798: [SPARK-27724][SQL]
Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#discussion_r305604944
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
##
@@ -89,15 +92,145 @@ case class CreateTableAsSelectExec(
case _ =>
// table does not support writes
- throw new SparkException(s"Table implementation does not support
writes: ${ident.quoted}")
+ throw new SparkException(
+s"Table implementation does not support writes: ${ident.quoted}")
}
+})(catchBlock = {
+ catalog.dropTable(ident)
+})
+ }
+}
+
+/**
+ * Physical plan node for v2 create table as select, when the catalog is
determined to support
+ * staging table creation.
+ *
+ * A new table will be created using the schema of the query, and rows from
the query are appended.
+ * The CTAS operation is atomic. The creation of the table is staged and the
commit of the write
+ * should bundle the commitment of the metadata and the table contents in a
single unit. If the
+ * write fails, the table is instructed to roll back all staged changes.
+ */
+case class AtomicCreateTableAsSelectExec(
+catalog: StagingTableCatalog,
+ident: Identifier,
+partitioning: Seq[Transform],
+query: SparkPlan,
+properties: Map[String, String],
+writeOptions: CaseInsensitiveStringMap,
+ifNotExists: Boolean) extends AtomicTableWriteExec {
+
+ override protected def doExecute(): RDD[InternalRow] = {
+if (catalog.tableExists(ident)) {
+ if (ifNotExists) {
+return sparkContext.parallelize(Seq.empty, 1)
+ }
+
+ throw new TableAlreadyExistsException(ident)
+}
+val stagedTable = catalog.stageCreate(
+ ident, query.schema, partitioning.toArray, properties.asJava)
+writeToStagedTable(stagedTable, writeOptions, ident)
+ }
+}
+
+/**
+ * Physical plan node for v2 replace table as select when the catalog does not
support staging
+ * table replacement.
+ *
+ * A new table will be created using the schema of the query, and rows from
the query are appended.
+ * If the table exists, its contents and schema should be replaced with the
schema and the contents
+ * of the query. This is a non-atomic implementation that drops the table and
then runs non-atomic
+ * CTAS. For an atomic implementation for catalogs with the appropriate
support, see
+ * ReplaceTableAsSelectStagingExec.
+ */
+case class ReplaceTableAsSelectExec(
+catalog: TableCatalog,
+ident: Identifier,
+partitioning: Seq[Transform],
+query: SparkPlan,
+properties: Map[String, String],
+writeOptions: CaseInsensitiveStringMap,
+orCreate: Boolean) extends AtomicTableWriteExec {
+
+ import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.IdentifierHelper
+
+ override protected def doExecute(): RDD[InternalRow] = {
+// Note that this operation is potentially unsafe, but these are the
strict semantics of
+// RTAS if the catalog does not support atomic operations.
+//
+// There are numerous cases we concede to where the table will be dropped
and irrecoverable:
+//
+// 1. Creating the new table fails,
+// 2. Writing to the new table fails,
+// 3. The table returned by catalog.createTable doesn't support writing.
+if (catalog.tableExists(ident)) {
+ catalog.dropTable(ident)
+} else if (!orCreate) {
+ throw new CannotReplaceMissingTableException(ident)
+}
+val createdTable = catalog.createTable(
+ ident, query.schema, partitioning.toArray, properties.asJava)
+Utils.tryWithSafeFinallyAndFailureCallbacks({
+ createdTable match {
+case table: SupportsWrite =>
+ val batchWrite = table.newWriteBuilder(writeOptions)
+.withInputDataSchema(query.schema)
+.withQueryId(UUID.randomUUID().toString)
+.buildForBatch()
+
+ doWrite(batchWrite)
+case _ =>
+ // table does not support writes
+ throw new SparkException(
+s"Table implementation does not support writes: ${ident.quoted}")
+ }
})(catchBlock = {
catalog.dropTable(ident)
})
}
}
+/**
+ *
+ * Physical plan node for v2 replace table as select when the catalog supports
staging
+ * table replacement.
+ *
+ * A new table will be created using the schema of the query, and rows from
the query are appended.
+ * If the table exists, its contents and schema should be replaced with the
schema and the contents
+ * of the query. This implementation is atomic. The table replacement is
staged, and the commit
+ * operation at the end should perform tne replacement of the table's metadata
and contents. If the
+ * write fails, the table is instructed to roll back staged changes and any
previously written table
+ * is left untouched.
+ */
+case
[GitHub] [spark] MaxGekk commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
MaxGekk commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305605279 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1605,3 +1605,49 @@ private case class GetTimestamp( override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression = copy(timeZoneId = Option(timeZoneId)) } + +@ExpressionDescription( + usage = "_FUNC_(year, month, day) - Create date from year, month and day fields.", + arguments = """ +Arguments: + * year - the year to represent, from 1 to + * month - the month-of-year to represent, from 1 (January) to 12 (December) + * day - the day-of-month to represent, from 1 to 31 + """, + examples = """ +Examples: + > SELECT _FUNC_(2013, 7, 15); + '2013-07-15' Review comment: I removed the quotes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25111: [SPARK-28346][SQL] clone the query plan between analyzer, optimizer and planner
cloud-fan commented on a change in pull request #25111: [SPARK-28346][SQL]
clone the query plan between analyzer, optimizer and planner
URL: https://github.com/apache/spark/pull/25111#discussion_r305605605
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala
##
@@ -137,5 +139,38 @@ class QueryExecutionSuite extends SharedSQLContext {
(_: LogicalPlan) => throw new Error("error"))
val error = intercept[Error](qe.toString)
assert(error.getMessage.contains("error"))
+
+spark.experimental.extraStrategies = Nil
+ }
+
+ test("SPARK-28346: clone the query plan between analyzer, optimizer and
planner") {
Review comment:
This test fails in the latest master branch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305605866 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1605,3 +1605,49 @@ private case class GetTimestamp( override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression = copy(timeZoneId = Option(timeZoneId)) } + +@ExpressionDescription( + usage = "_FUNC_(year, month, day) - Create date from year, month and day fields.", + arguments = """ +Arguments: + * year - the year to represent, from 1 to + * month - the month-of-year to represent, from 1 (January) to 12 (December) + * day - the day-of-month to represent, from 1 to 31 + """, + examples = """ +Examples: + > SELECT _FUNC_(2013, 7, 15); + '2013-07-15' Review comment: Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mgaido91 commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
mgaido91 commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-513532789 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
SparkQA commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-513532886 **[Test build #107960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107960/testReport)** for PR 21599 at commit [`650ea79`](https://github.com/apache/spark/commit/650ea796dafa5115f49115042b409e300ebc79de). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API
cloud-fan commented on issue #25206: [SPARK-28265][SQL] Add renameTable to TableCatalog API URL: https://github.com/apache/spark/pull/25206#issuecomment-513532914 > but we do need to decide whether to require that both identifiers start with the same catalog A similar problem was already discussed before and the decision was: RENAME TABLE should only do rename, not things like moving databases. As a result, the `ExternalCatalog.renameTable` is defined as `def renameTable(db: String, oldName: String, newName: String): Unit`. We can also enforce it at the parser level: `ALTER TABLE multiPartName RENAME TO singlePartName`. We didn't do it before for backward compatibility reasons, it's a good chance to do it for DS v2 now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513533256 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513533258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13069/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513533258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13069/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function
AmplabJenkins removed a comment on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513533256 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-513533284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-513533285 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13070/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-513533285 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13070/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function
SparkQA commented on issue #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#issuecomment-513533391 **[Test build #107961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107961/testReport)** for PR 25210 at commit [`30e62c2`](https://github.com/apache/spark/commit/30e62c207bbd0bde7987773ae93e44eb26ffeb1c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-513533284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
dongjoon-hyun commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533525 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513522360 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
MaxGekk commented on a change in pull request #25210: [SPARK-28432][SQL] Add
`make_date` function
URL: https://github.com/apache/spark/pull/25210#discussion_r305606571
##
File path: sql/core/src/test/resources/sql-tests/results/pgSQL/date.sql.out
##
@@ -508,8 +508,48 @@ struct
-- !query 47
-DROP TABLE DATE_TBL
+select make_date(2013, 7, 15)
-- !query 47 schema
-struct<>
+struct
-- !query 47 output
+2013-07-15
+
+
+-- !query 48
+select make_date(-44, 3, 15)
+-- !query 48 schema
+struct
+-- !query 48 output
+0045-03-15
Review comment:
The year `-44` is out of valid range according to SQL standard. We are
getting `45` instead of `-44` while converting to java.sql.Date. If you switch
to Java 8 API for date/timestamps:
```Scala
scala> spark.conf.set("spark.sql.datetime.java8API.enabled", true)
scala> spark.sql("select make_date(-44, 3, 15)").collect
res7: Array[org.apache.spark.sql.Row] = Array([-0044-03-15])
```
the returned instance of `java.time.LocalDate` seems reasonable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533781 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13071/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533780 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#discussion_r305606621 ## File path: docs/structured-streaming-kafka-integration.md ## @@ -388,6 +388,19 @@ The following configurations are optional: streaming and batch Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume. + + minPartitions + int + 0 (disabled) Review comment: Thank you for your first contribution, @arunpandianp . However, this is wrong because this will mislead the users to try to set `0` and face `IllegalArgumentException`. Technically, the default value is `None`. Just leave this line as a blank like ``. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533951 **[Test build #107962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107962/testReport)** for PR 25219 at commit [`69683eb`](https://github.com/apache/spark/commit/69683eba1d38947bad307d3dadf716b028b5c270). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533780 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533781 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13071/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305606883 ## File path: sql/core/src/test/resources/sql-tests/results/pgSQL/date.sql.out ## @@ -508,8 +508,48 @@ struct -- !query 47 -DROP TABLE DATE_TBL +select make_date(2013, 7, 15) -- !query 47 schema -struct<> +struct -- !query 47 output +2013-07-15 + + +-- !query 48 +select make_date(-44, 3, 15) +-- !query 48 schema +struct +-- !query 48 output +0045-03-15 Review comment: You need to file a JIRA issue for this difference on `make_date` input range checking. Also, please add the JIRA id to `date.sql` file. FYI, the following is PostgreSQL output. - https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/expected/date.out This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305606915 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -351,13 +351,13 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- skip test constructors Review comment: Please remove line 352, too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305606977 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -351,13 +351,13 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- skip test constructors -- test constructors --- select make_date(2013, 7, 15); --- select make_date(-44, 3, 15); +select make_date(2013, 7, 15); +select make_date(-44, 3, 15); -- select make_time(8, 20, 0.0); Review comment: If you want to keep this as a comment. You need to add JIRA issue above this line. The following is the PostgreSQL result. ``` select make_time(8, 20, 0.0); make_time --- 08:20:00 (1 row) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305606998 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -351,13 +351,13 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- skip test constructors -- test constructors --- select make_date(2013, 7, 15); --- select make_date(-44, 3, 15); +select make_date(2013, 7, 15); +select make_date(-44, 3, 15); -- select make_time(8, 20, 0.0); -- should fail --- select make_date(2013, 2, 30); --- select make_date(2013, 13, 1); --- select make_date(2013, 11, -1); +select make_date(2013, 2, 30); +select make_date(2013, 13, 1); +select make_date(2013, 11, -1); -- select make_time(10, 55, 100.1); -- select make_time(24, 0, 2.1); Review comment: ditto. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305607042 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -351,13 +351,13 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- skip test constructors -- test constructors --- select make_date(2013, 7, 15); --- select make_date(-44, 3, 15); +select make_date(2013, 7, 15); +select make_date(-44, 3, 15); -- select make_time(8, 20, 0.0); Review comment: Oops. Please ignore the above. It's `make_time`. I was confused. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305606998 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -351,13 +351,13 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- skip test constructors -- test constructors --- select make_date(2013, 7, 15); --- select make_date(-44, 3, 15); +select make_date(2013, 7, 15); +select make_date(-44, 3, 15); -- select make_time(8, 20, 0.0); -- should fail --- select make_date(2013, 2, 30); --- select make_date(2013, 13, 1); --- select make_date(2013, 11, -1); +select make_date(2013, 2, 30); +select make_date(2013, 13, 1); +select make_date(2013, 11, -1); -- select make_time(10, 55, 100.1); -- select make_time(24, 0, 2.1); Review comment: ditto. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function
dongjoon-hyun commented on a change in pull request #25210: [SPARK-28432][SQL] Add `make_date` function URL: https://github.com/apache/spark/pull/25210#discussion_r305606977 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql ## @@ -351,13 +351,13 @@ SELECT f1 - date '2000-01-01' AS `Days From 2K` FROM DATE_TBL; -- skip test constructors -- test constructors --- select make_date(2013, 7, 15); --- select make_date(-44, 3, 15); +select make_date(2013, 7, 15); +select make_date(-44, 3, 15); -- select make_time(8, 20, 0.0); Review comment: ~If you want to keep this as a comment. You need to add JIRA issue above this line. The following is the PostgreSQL result.~ ``` select make_time(8, 20, 0.0); make_time --- 08:20:00 (1 row) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] arunpandianp commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
arunpandianp commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513534549 @dongjoon-hyun thanks for checking, changed it to `` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513534758 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513534760 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107962/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
SparkQA removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513533951 **[Test build #107962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107962/testReport)** for PR 25219 at commit [`69683eb`](https://github.com/apache/spark/commit/69683eba1d38947bad307d3dadf716b028b5c270). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #24228: [SPARK-27280][SQL]infer more filter from join or condition
maropu commented on issue #24228: [SPARK-27280][SQL]infer more filter from join or condition URL: https://github.com/apache/spark/pull/24228#issuecomment-513534765 I also think we need not a band-aid fix, but a general solution for this issue as @cloud-fan suggested in #24973. Have you checked the previous activity for the CNF normalization? https://issues.apache.org/jira/browse/SPARK-6624 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513534734 **[Test build #107962 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107962/testReport)** for PR 25219 at commit [`69683eb`](https://github.com/apache/spark/commit/69683eba1d38947bad307d3dadf716b028b5c270). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513534758 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513534760 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107962/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513535036 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13072/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513535033 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513535036 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13072/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins removed a comment on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513535033 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#discussion_r305607369 ## File path: docs/structured-streaming-kafka-integration.md ## @@ -388,6 +388,19 @@ The following configurations are optional: streaming and batch Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume. + + minPartitions + int + + streaming and batch + Minimum number of partitions to read from Kafka. + You can configure Spark to use an arbitrary minimum of partitions to read from Kafka using the minPartitions option. Review comment: Let's remove this line because we don't allow `arbitrary` number. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#discussion_r305607379 ## File path: docs/structured-streaming-kafka-integration.md ## @@ -388,6 +388,19 @@ The following configurations are optional: streaming and batch Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume. + + minPartitions + int + + streaming and batch + Minimum number of partitions to read from Kafka. + You can configure Spark to use an arbitrary minimum of partitions to read from Kafka using the minPartitions option. + Normally Spark has a 1-1 mapping of Kafka TopicPartitions to Spark partitions consuming from Kafka. Review comment: `Normally` -> `By default, `? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
dongjoon-hyun commented on a change in pull request #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#discussion_r305607418 ## File path: docs/structured-streaming-kafka-integration.md ## @@ -388,6 +388,19 @@ The following configurations are optional: streaming and batch Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume. + + minPartitions + int + + streaming and batch + Minimum number of partitions to read from Kafka. + You can configure Spark to use an arbitrary minimum of partitions to read from Kafka using the minPartitions option. + Normally Spark has a 1-1 mapping of Kafka TopicPartitions to Spark partitions consuming from Kafka. + If you set the minPartitions option to a value greater than your Kafka TopicPartitions, + Spark will divvy up large Kafka partitions to smaller pieces. + This option can be set at times of peak loads, data skew, and as your stream is falling behind to increase processing rate. + It comes at a cost of initializing Kafka consumers at each trigger, which may impact performance if you use SSL when connecting to Kafka. Review comment: Let's remove line 401~402, too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513535189 **[Test build #107963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107963/testReport)** for PR 25219 at commit [`6081c50`](https://github.com/apache/spark/commit/6081c50cdb7ea7b1383938b0f2b8ef57bfbb4cfd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
SparkQA commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513535995 **[Test build #107963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107963/testReport)** for PR 25219 at commit [`6081c50`](https://github.com/apache/spark/commit/6081c50cdb7ea7b1383938b0f2b8ef57bfbb4cfd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option
AmplabJenkins commented on issue #25219: [SPARK-28464][Doc][SS] Document Kafka source minPartitions option URL: https://github.com/apache/spark/pull/25219#issuecomment-513536019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107963/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
