[GitHub] [spark] AmplabJenkins commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
AmplabJenkins commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365#issuecomment-518515614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13784/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
AmplabJenkins commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365#issuecomment-518515608 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
SparkQA commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog URL: https://github.com/apache/spark/pull/25363#issuecomment-518515549 **[Test build #108692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108692/testReport)** for PR 25363 at commit [`b4f3a93`](https://github.com/apache/spark/commit/b4f3a93b06b07c79236577cec1e58170bd0f6969). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class V2SessionCatalog(sessionState: SessionState) extends TableCatalog with SupportsNamespaces ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518515346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518509713 **[Test build #108697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108697/testReport)** for PR 25228 at commit [`1c9afae`](https://github.com/apache/spark/commit/1c9afae88de80a69c872488d520574f37ff95ec3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518515356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518515346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518515356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
HyukjinKwon commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365#issuecomment-518515458 Thanks for quick review and sign-off! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518515261 **[Test build #108697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108697/testReport)** for PR 25228 at commit [`1c9afae`](https://github.com/apache/spark/commit/1c9afae88de80a69c872488d520574f37ff95ec3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
sarutak commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365#issuecomment-518515039 I've also come up with the same solution with you. LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
HyukjinKwon commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365#issuecomment-518514252 cc @cloud-fan, @maropu, and @sarutak This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
SparkQA commented on issue #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365#issuecomment-518513993 **[Test build #108698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108698/testReport)** for PR 25365 at commit [`352a3cb`](https://github.com/apache/spark/commit/352a3cb40c851cdba5e4289095d54809438f0a7b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder
AmplabJenkins removed a comment on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder URL: https://github.com/apache/spark/pull/24983#issuecomment-518513254 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108691/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
SparkQA removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518507457 **[Test build #108696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108696/testReport)** for PR 25228 at commit [`b44804a`](https://github.com/apache/spark/commit/b44804a43fafbc1756c076003c131d7274d512c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518513381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518513377 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec
HyukjinKwon opened a new pull request #25365: [SPARK-28537][SQL][HOTFIX][FOLLOW-UP] Add supportColumnar in DebugExec URL: https://github.com/apache/spark/pull/25365 ## What changes were proposed in this pull request? This PR add supportColumnar in DebugExec. Seems there was a conflict between https://github.com/apache/spark/pull/25274 and https://github.com/apache/spark/pull/25264 Currently tests are broken in Jenkins: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108687/ https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108688/ https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108693/ ## How was this patch tested? Manually tested the failed test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518513381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder
AmplabJenkins removed a comment on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder URL: https://github.com/apache/spark/pull/24983#issuecomment-518513250 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518513377 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder
AmplabJenkins commented on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder URL: https://github.com/apache/spark/pull/24983#issuecomment-518513254 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108691/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518513274 **[Test build #108696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108696/testReport)** for PR 25228 at commit [`b44804a`](https://github.com/apache/spark/commit/b44804a43fafbc1756c076003c131d7274d512c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder
AmplabJenkins commented on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder URL: https://github.com/apache/spark/pull/24983#issuecomment-518513250 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder
SparkQA removed a comment on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder URL: https://github.com/apache/spark/pull/24983#issuecomment-518479523 **[Test build #108691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108691/testReport)** for PR 24983 at commit [`dcb9771`](https://github.com/apache/spark/commit/dcb977178bd11a9662bf5c62baa478c3fa949cfe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder
SparkQA commented on issue #24983: [SPARK-27714][SQL][CBO] Support Genetic Algorithm based join reorder URL: https://github.com/apache/spark/pull/24983#issuecomment-518513007 **[Test build #108691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108691/testReport)** for PR 24983 at commit [`dcb9771`](https://github.com/apache/spark/commit/dcb977178bd11a9662bf5c62baa478c3fa949cfe). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25274: [SPARK-28537][SQL] DebugExec cannot debug broadcast or columnar related queries.
HyukjinKwon commented on issue #25274: [SPARK-28537][SQL] DebugExec cannot debug broadcast or columnar related queries. URL: https://github.com/apache/spark/pull/25274#issuecomment-518510451 seems https://github.com/apache/spark/commit/03e30063127fd71bef8a14553381e805fe5b6679 was conflicted with this one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25274: [SPARK-28537][SQL] DebugExec cannot debug broadcast or columnar related queries.
HyukjinKwon commented on issue #25274: [SPARK-28537][SQL] DebugExec cannot debug broadcast or columnar related queries. URL: https://github.com/apache/spark/pull/25274#issuecomment-518510159 Hmmm .. seems this commit causes the test failure: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108687/ https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108688/ https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108693/ There are three consecutive builds being failed and it's failed in my local too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518509713 **[Test build #108697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108697/testReport)** for PR 25228 at commit [`1c9afae`](https://github.com/apache/spark/commit/1c9afae88de80a69c872488d520574f37ff95ec3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518509120 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
AmplabJenkins removed a comment on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file URL: https://github.com/apache/spark/pull/25357#issuecomment-518509043 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
AmplabJenkins removed a comment on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file URL: https://github.com/apache/spark/pull/25357#issuecomment-518509048 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13782/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518509121 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13783/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518509120 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
AmplabJenkins commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file URL: https://github.com/apache/spark/pull/25357#issuecomment-518509043 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518509121 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13783/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
AmplabJenkins commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file URL: https://github.com/apache/spark/pull/25357#issuecomment-518509048 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13782/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
SparkQA commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518507457 **[Test build #108696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108696/testReport)** for PR 25228 at commit [`b44804a`](https://github.com/apache/spark/commit/b44804a43fafbc1756c076003c131d7274d512c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
SparkQA commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file URL: https://github.com/apache/spark/pull/25357#issuecomment-518507447 **[Test build #108695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108695/testReport)** for PR 25357 at commit [`eeb7405`](https://github.com/apache/spark/commit/eeb7405ad0c7cc1004e2cad36929d20d95ab2726). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518506865 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins removed a comment on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518506871 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13781/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518506865 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions
AmplabJenkins commented on issue #25228: [SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions URL: https://github.com/apache/spark/pull/25228#issuecomment-518506871 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13781/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25331: [SPARK-27768][SQL] Infinity, -Infinity, NaN should be recognized in a case insensitive manner.
dilipbiswal commented on a change in pull request #25331: [SPARK-27768][SQL]
Infinity, -Infinity, NaN should be recognized in a case insensitive manner.
URL: https://github.com/apache/spark/pull/25331#discussion_r310885528
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -562,8 +593,12 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
// FloatConverter
private[this] def castToFloat(from: DataType): Any => Any = from match {
case StringType =>
- buildCast[UTF8String](_, s => try s.toString.toFloat catch {
-case _: NumberFormatException => null
+ buildCast[UTF8String](_, s => {
+val floatStr = s.toString
+try floatStr.toFloat catch {
+ case _: NumberFormatException =>
Review comment:
@maropu Oops.. didn't see this comment. I suppose i have to use the
benchmark framework for this ? Appreciate any tip on this..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518505971 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108693/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518505965 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518505965 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #25327: [SPARK-28593][CORE] Rename ShuffleClient to BlockStoreClient which more close to its usage
xuanyuanking commented on issue #25327: [SPARK-28593][CORE] Rename ShuffleClient to BlockStoreClient which more close to its usage URL: https://github.com/apache/spark/pull/25327#issuecomment-518505797 Yep, thanks for the advice and review :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
SparkQA removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518487495 **[Test build #108693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108693/testReport)** for PR 24892 at commit [`eb5c58b`](https://github.com/apache/spark/commit/eb5c58b35427497accb43c47423c23c48810aa78). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518505971 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108693/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
SparkQA commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518505743 **[Test build #108693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108693/testReport)** for PR 24892 at commit [`eb5c58b`](https://github.com/apache/spark/commit/eb5c58b35427497accb43c47423c23c48810aa78). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
SparkQA commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518505387 **[Test build #108694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108694/testReport)** for PR 25355 at commit [`a07dce5`](https://github.com/apache/spark/commit/a07dce5a71b62d21064fc585f7ef746fb2fff6cc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518504830 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13780/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518504819 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518504830 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13780/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518504819 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
HyukjinKwon commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518503962 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r310882371
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +184,29 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * If MEMORY_OFFHEAP_ENABLED is true, we should ensure
executorOverheadMemory requested value
+ * is not less than MEMORY_OFFHEAP_SIZE, otherwise the memory resource
requested for executor
+ * may be not enough.
+ */
+ def executorMemoryOverheadRequested(sparkConf: SparkConf): Int = {
+val executorMemory = sparkConf.get(EXECUTOR_MEMORY).toInt
+val overhead = sparkConf.get(EXECUTOR_MEMORY_OVERHEAD).getOrElse(
+ math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt,
MEMORY_OVERHEAD_MIN)).toInt
+val offHeap = if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val size =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString)
+ require(size > 0,
+s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
+ if (size > overhead) {
+logWarning(s"The value of ${MEMORY_OFFHEAP_SIZE.key}(${size}MB) will
be used as " +
+ s"executorMemoryOverhead to request resource to ensure that Executor
has enough memory " +
+ s"to use. It is recommended that the configuration value of " +
+ s"${EXECUTOR_MEMORY_OVERHEAD.key} should be no less than
${MEMORY_OFFHEAP_SIZE.key} " +
+ s"when ${MEMORY_OFFHEAP_ENABLED.key} is true.")
+ }
+ size
+} else 0
+math.max(overhead, offHeap).toInt
Review comment:
@jerryshao Is the current approach feasible?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing
AmplabJenkins removed a comment on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing URL: https://github.com/apache/spark/pull/25328#issuecomment-518500345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108688/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing
AmplabJenkins removed a comment on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing URL: https://github.com/apache/spark/pull/25328#issuecomment-518500339 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing
AmplabJenkins commented on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing URL: https://github.com/apache/spark/pull/25328#issuecomment-518500345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108688/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing
AmplabJenkins commented on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing URL: https://github.com/apache/spark/pull/25328#issuecomment-518500339 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing
SparkQA removed a comment on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing URL: https://github.com/apache/spark/pull/25328#issuecomment-518470489 **[Test build #108688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108688/testReport)** for PR 25328 at commit [`0652c22`](https://github.com/apache/spark/commit/0652c224466be741f985b77104cfbebb2cbf1a9e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing
SparkQA commented on issue #25328: [SPARK-28595][SQL] explain should not trigger partition listing URL: https://github.com/apache/spark/pull/25328#issuecomment-518500137 **[Test build #108688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108688/testReport)** for PR 25328 at commit [`0652c22`](https://github.com/apache/spark/commit/0652c224466be741f985b77104cfbebb2cbf1a9e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518495924 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108687/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518495920 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518495924 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108687/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
SparkQA removed a comment on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518468827 **[Test build #108687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108687/testReport)** for PR 25355 at commit [`a07dce5`](https://github.com/apache/spark/commit/a07dce5a71b62d21064fc585f7ef746fb2fff6cc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
AmplabJenkins commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518495920 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default
SparkQA commented on issue #25355: [SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default URL: https://github.com/apache/spark/pull/25355#issuecomment-518495786 **[Test build #108687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108687/testReport)** for PR 25355 at commit [`a07dce5`](https://github.com/apache/spark/commit/a07dce5a71b62d21064fc585f7ef746fb2fff6cc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
maropu commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-518495040 @dongjoon-hyun could you check this pr? (I have no comment now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
maropu commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-518494385 > @maropu Could I open another PR to resolve this? Since this jira originally intends to support these statements with escape, I think its ok to uncomment them in this pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
maropu commented on a change in pull request #25357: [SPARK-28617][SQL][TEST]
Completely remove comments from the golden result file
URL: https://github.com/apache/spark/pull/25357#discussion_r310876905
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -224,7 +224,8 @@ class SQLQueryTestSuite extends QueryTest with
SharedSQLContext {
private def runTest(testCase: TestCase): Unit = {
val input = fileToString(new File(testCase.inputFile))
-val (comments, code) = input.split("\n").partition(_.startsWith("--"))
+val (comments, codeMaybeWithComment) =
input.split("\n").partition(_.trim.startsWith("--"))
+val code = codeMaybeWithComment.map(_.split("--").head)
Review comment:
oh... really weird output... nice catch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r310876754
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +184,29 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * If MEMORY_OFFHEAP_ENABLED is true, we should ensure
executorOverheadMemory requested value
+ * is not less than MEMORY_OFFHEAP_SIZE, otherwise the memory resource
requested for executor
+ * may be not enough.
+ */
+ def executorMemoryOverheadRequested(sparkConf: SparkConf): Int = {
+val executorMemory = sparkConf.get(EXECUTOR_MEMORY).toInt
+val overhead = sparkConf.get(EXECUTOR_MEMORY_OVERHEAD).getOrElse(
+ math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt,
MEMORY_OVERHEAD_MIN)).toInt
+val offHeap = if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val size =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString)
+ require(size > 0,
+s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
+ if (size > overhead) {
+logWarning(s"The value of ${MEMORY_OFFHEAP_SIZE.key}(${size}MB) will
be used as " +
+ s"executorMemoryOverhead to request resource to ensure that Executor
has enough memory " +
+ s"to use. It is recommended that the configuration value of " +
+ s"${EXECUTOR_MEMORY_OVERHEAD.key} should be no less than
${MEMORY_OFFHEAP_SIZE.key} " +
+ s"when ${MEMORY_OFFHEAP_ENABLED.key} is true.")
+ }
+ size
+} else 0
+math.max(overhead, offHeap).toInt
Review comment:
On the other hand, if the user configures `offheapMemory` and
`pysparkWorkerMemory`,
He still needs to configure overhead Memroy and ensure that the
configuration is reasonable(` memoryOverhead > offheapMemory +
pysparkWorkerMemory `) in Yarn mode, so that users may need to care about more
details.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file
maropu commented on issue #25357: [SPARK-28617][SQL][TEST] Completely remove comments from the golden result file URL: https://github.com/apache/spark/pull/25357#issuecomment-518491485 +1, too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r310875078
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +184,29 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * If MEMORY_OFFHEAP_ENABLED is true, we should ensure
executorOverheadMemory requested value
+ * is not less than MEMORY_OFFHEAP_SIZE, otherwise the memory resource
requested for executor
+ * may be not enough.
+ */
+ def executorMemoryOverheadRequested(sparkConf: SparkConf): Int = {
+val executorMemory = sparkConf.get(EXECUTOR_MEMORY).toInt
+val overhead = sparkConf.get(EXECUTOR_MEMORY_OVERHEAD).getOrElse(
+ math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt,
MEMORY_OVERHEAD_MIN)).toInt
+val offHeap = if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val size =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString)
+ require(size > 0,
+s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
+ if (size > overhead) {
+logWarning(s"The value of ${MEMORY_OFFHEAP_SIZE.key}(${size}MB) will
be used as " +
+ s"executorMemoryOverhead to request resource to ensure that Executor
has enough memory " +
+ s"to use. It is recommended that the configuration value of " +
+ s"${EXECUTOR_MEMORY_OVERHEAD.key} should be no less than
${MEMORY_OFFHEAP_SIZE.key} " +
+ s"when ${MEMORY_OFFHEAP_ENABLED.key} is true.")
+ }
+ size
+} else 0
+math.max(overhead, offHeap).toInt
Review comment:
@beliefer Now `YarnAllocator` line 150 use `executorMemory + memoryOverhead
+ pysparkWorkerMemory` to new `Resource` Instance, Is this wrong?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r310875078
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +184,29 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * If MEMORY_OFFHEAP_ENABLED is true, we should ensure
executorOverheadMemory requested value
+ * is not less than MEMORY_OFFHEAP_SIZE, otherwise the memory resource
requested for executor
+ * may be not enough.
+ */
+ def executorMemoryOverheadRequested(sparkConf: SparkConf): Int = {
+val executorMemory = sparkConf.get(EXECUTOR_MEMORY).toInt
+val overhead = sparkConf.get(EXECUTOR_MEMORY_OVERHEAD).getOrElse(
+ math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt,
MEMORY_OVERHEAD_MIN)).toInt
+val offHeap = if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val size =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString)
+ require(size > 0,
+s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
+ if (size > overhead) {
+logWarning(s"The value of ${MEMORY_OFFHEAP_SIZE.key}(${size}MB) will
be used as " +
+ s"executorMemoryOverhead to request resource to ensure that Executor
has enough memory " +
+ s"to use. It is recommended that the configuration value of " +
+ s"${EXECUTOR_MEMORY_OVERHEAD.key} should be no less than
${MEMORY_OFFHEAP_SIZE.key} " +
+ s"when ${MEMORY_OFFHEAP_ENABLED.key} is true.")
+ }
+ size
+} else 0
+math.max(overhead, offHeap).toInt
Review comment:
@beliefer Now `YarnAllocator` line 150 use `executorMemory + memoryOverhead
+ pysparkWorkerMemory` to new `Resource` Instance, Is this wrong?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518488448 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins removed a comment on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518488452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13779/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518488448 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518488452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13779/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
SparkQA commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518487495 **[Test build #108693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108693/testReport)** for PR 24892 at commit [`eb5c58b`](https://github.com/apache/spark/commit/eb5c58b35427497accb43c47423c23c48810aa78). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #25361: [SPARK-28625][Core] Indeterminate shuffle support in Shuffle Writer API
xuanyuanking commented on a change in pull request #25361: [SPARK-28625][Core] Indeterminate shuffle support in Shuffle Writer API URL: https://github.com/apache/spark/pull/25361#discussion_r310872522 ## File path: core/src/main/java/org/apache/spark/shuffle/api/ShuffleExecutorComponents.java ## @@ -39,16 +39,20 @@ /** * Called once per map task to create a writer that will be responsible for persisting all the * partitioned bytes written by that map task. - * @param shuffleId Unique identifier for the shuffle the map task is a part of + * @param shuffleId Unique identifier for the shuffle the map task is a part of + * @param shuffleGenerationId The shuffle generation ID of the stage that this task belongs to, Review comment: Yep, thanks for your faster review, I describe the requirement in https://github.com/apache/spark/pull/24892#issuecomment-518487347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
xuanyuanking commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-518487347 The task attempt id is not enough in this situation, because for the indeterminate stage, while a single task fails, we need a whole stage rerun, only retry the failure task will cause correctness bug, you may find more details in #22112. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
AmplabJenkins removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364#issuecomment-518487020 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
AmplabJenkins removed a comment on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364#issuecomment-518486149 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
AmplabJenkins commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364#issuecomment-518487093 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25218: [SPARK-28466][SQL] fix sts FileSystem closed error when to call Hive.moveFile
AngersZh commented on issue #25218: [SPARK-28466][SQL] fix sts FileSystem closed error when to call Hive.moveFile URL: https://github.com/apache/spark/pull/25218#issuecomment-518487091 > Hi @AngersZh , is this similar to [[SPARK-21067]](https://issues.apache.org/jira/browse/SPARK-21067)? Yes, in this pr's description, youc an know the root cause about that error. But my method is not good enough. If you just use Hive1.2.1 ， below PR is ok https://github.com/apache/spark/pull/25197 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
AmplabJenkins commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364#issuecomment-518487020 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25335: [SPARK-28601][CORE][SQL] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException
dongjoon-hyun closed pull request #25335: [SPARK-28601][CORE][SQL] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
Deegue commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364#issuecomment-518486705 Links to #25197 #25058 #25218 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter
dongjoon-hyun commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#discussion_r310871561 ## File path: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ## @@ -670,11 +671,9 @@ private[spark] class ExternalSorter[K, V, C]( } /** - * Write all the data added into this ExternalSorter into a file in the disk store. This is - * called by the SortShuffleWriter. - * - * @param blockId block ID to write to. The index file will be blockId.name + ".index". - * @return array of lengths, in bytes, of each partition of the file (used by map output tracker) + * TODO remove this, as this is only used by UnsafeRowSerializerSuite in the SQL project. Review comment: nit. Could you file a JIRA and make this IDed TODO, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
AmplabJenkins commented on issue #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364#issuecomment-518486149 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue opened a new pull request #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source
Deegue opened a new pull request #25364: [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move source URL: https://github.com/apache/spark/pull/25364 ## What changes were proposed in this pull request? This PR aims to fix CTAS fails after we closed a session of ThriftServer. - sql-distributed-sql-engine.md  It seems the simplest way to fix [[SPARK-21067]](https://issues.apache.org/jira/browse/SPARK-21067). ## How was this patch tested Manual. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue commented on issue #25218: [SPARK-28466][SQL] fix sts FileSystem closed error when to call Hive.moveFile
Deegue commented on issue #25218: [SPARK-28466][SQL] fix sts FileSystem closed error when to call Hive.moveFile URL: https://github.com/apache/spark/pull/25218#issuecomment-518485338 Hi @AngersZh , is this similar to [[SPARK-21067]](https://issues.apache.org/jira/browse/SPARK-21067)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
SparkQA commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog URL: https://github.com/apache/spark/pull/25363#issuecomment-518483679 **[Test build #108692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108692/testReport)** for PR 25363 at commit [`b4f3a93`](https://github.com/apache/spark/commit/b4f3a93b06b07c79236577cec1e58170bd0f6969). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
AmplabJenkins removed a comment on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog URL: https://github.com/apache/spark/pull/25363#issuecomment-518483312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13778/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
AmplabJenkins removed a comment on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog URL: https://github.com/apache/spark/pull/25363#issuecomment-518483307 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
AmplabJenkins commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog URL: https://github.com/apache/spark/pull/25363#issuecomment-518483312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13778/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
AmplabJenkins commented on issue #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog URL: https://github.com/apache/spark/pull/25363#issuecomment-518483307 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
