[GitHub] [spark] dongjoon-hyun commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
dongjoon-hyun commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631254257 Thank you, @HyukjinKwon . Since this helps the testing infra, I'll backport to branch-3.0 too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
HyukjinKwon closed pull request #28587: URL: https://github.com/apache/spark/pull/28587 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
HyukjinKwon commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631253486 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
HyukjinKwon commented on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631249658 Thank you @dongjoon-hyun. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631248730 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122866/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
SparkQA removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631205985 **[Test build #122866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122866/testReport)** for PR 28585 at commit [`f53735e`](https://github.com/apache/spark/commit/f53735e2e52a8cf3d70f162c3317c10383ae1aff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631248727 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631248727 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
SparkQA commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631248382 **[Test build #122866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122866/testReport)** for PR 28585 at commit [`f53735e`](https://github.com/apache/spark/commit/f53735e2e52a8cf3d70f162c3317c10383ae1aff). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
dongjoon-hyun closed pull request #28589: URL: https://github.com/apache/spark/pull/28589 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28527: [SPARK-31709][SQL] Proper base path for database/table location when it is a relative path
AmplabJenkins removed a comment on pull request #28527: URL: https://github.com/apache/spark/pull/28527#issuecomment-631247365 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28527: [SPARK-31709][SQL] Proper base path for database/table location when it is a relative path
AmplabJenkins commented on pull request #28527: URL: https://github.com/apache/spark/pull/28527#issuecomment-631247365 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28527: [SPARK-31709][SQL] Proper base path for database/table location when it is a relative path
SparkQA commented on pull request #28527: URL: https://github.com/apache/spark/pull/28527#issuecomment-631246654 **[Test build #122859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122859/testReport)** for PR 28527 at commit [`3fbe65a`](https://github.com/apache/spark/commit/3fbe65a0c3f1e59e5af1f5d3f3b7beb13c0636f6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28527: [SPARK-31709][SQL] Proper base path for database/table location when it is a relative path
SparkQA removed a comment on pull request #28527: URL: https://github.com/apache/spark/pull/28527#issuecomment-631146068 **[Test build #122859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122859/testReport)** for PR 28527 at commit [`3fbe65a`](https://github.com/apache/spark/commit/3fbe65a0c3f1e59e5af1f5d3f3b7beb13c0636f6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
AmplabJenkins removed a comment on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631241387 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122865/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
AmplabJenkins commented on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631241380 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
AmplabJenkins removed a comment on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631241380 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
AmplabJenkins removed a comment on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631241298 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
SparkQA removed a comment on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631197540 **[Test build #122865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122865/testReport)** for PR 28584 at commit [`b08a272`](https://github.com/apache/spark/commit/b08a27244934d498908bf50e85457b1a6a9ea099). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
AmplabJenkins commented on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631241298 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
SparkQA removed a comment on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631229769 **[Test build #122872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122872/testReport)** for PR 28589 at commit [`b4d215e`](https://github.com/apache/spark/commit/b4d215e9c4a8d6f336b6edc5ed6dce902c3debb5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
SparkQA commented on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631241046 **[Test build #122865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122865/testReport)** for PR 28584 at commit [`b08a272`](https://github.com/apache/spark/commit/b08a27244934d498908bf50e85457b1a6a9ea099). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
SparkQA commented on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631240905 **[Test build #122872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122872/testReport)** for PR 28589 at commit [`b4d215e`](https://github.com/apache/spark/commit/b4d215e9c4a8d6f336b6edc5ed6dce902c3debb5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
AmplabJenkins removed a comment on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631240376 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
SparkQA commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631240358 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27511/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
AmplabJenkins commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631240376 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28590: [SPARK-31768][ML] add getMetrics in Evaluators
AmplabJenkins removed a comment on pull request #28590: URL: https://github.com/apache/spark/pull/28590#issuecomment-631237933 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28590: [SPARK-31768][ML] add getMetrics in Evaluators
AmplabJenkins commented on pull request #28590: URL: https://github.com/apache/spark/pull/28590#issuecomment-631237933 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28590: [SPARK-31768][ML] add getMetrics in Evaluators
SparkQA commented on pull request #28590: URL: https://github.com/apache/spark/pull/28590#issuecomment-631237557 **[Test build #122873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122873/testReport)** for PR 28590 at commit [`e1a0a31`](https://github.com/apache/spark/commit/e1a0a316fca9138a03b3f79fa5f9f2805eabcbe0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao opened a new pull request #28590: [SPARK-31768][ML] add getMetrics in Evaluators
huaxingao opened a new pull request #28590:
URL: https://github.com/apache/spark/pull/28590
### What changes were proposed in this pull request?
add getMetrics in Evaluators to get the corresponding Metrics instance, so
users can use it to get any of the metrics scores. For example:
```
val trainer = new LinearRegression
val model = trainer.fit(dataset)
val predictions = model.transform(dataset)
val evaluator = new RegressionEvaluator()
val metrics = evaluator.getMetrics(predictions)
val rmse = metrics.rootMeanSquaredError
val r2 = metrics.r2
val mae = metrics.meanAbsoluteError
val variance = metrics.explainedVariance
```
### Why are the changes needed?
Currently, Evaluator.evaluate only access to one metrics, but most users may
need to get multiple metrics. This PR adds getMetrics in all the Evaluators, so
users can use it to get an instance of the corresponding Metrics to get any of
the metrics they want.
### Does this PR introduce _any_ user-facing change?
Yes. Add getMetrics in Evaluators.
For example:
```
/**
* Get a RegressionMetrics, which can be used to get any of the regression
* metrics such as rootMeanSquaredError, meanSquaredError, etc.
*
* @param dataset a dataset that contains labels/observations and
predictions.
* @return RegressionMetrics
*/
@Since("3.1.0")
def getMetrics(dataset: Dataset[_]): RegressionMetrics
```
### How was this patch tested?
Add new unit tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
dongjoon-hyun commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631236882 Could you review this PR, @HyukjinKwon ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
cloud-fan commented on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631235208 It's unfortunate that we find problems in SQL REST API at this stage which is too late to change. LGTM to revert it from 3.0 and release it in 3.1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
SparkQA commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631232801 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27511/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
HyukjinKwon closed pull request #28586: URL: https://github.com/apache/spark/pull/28586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631231179 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
HyukjinKwon commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631231141 Merged to branch-3.0. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
SparkQA removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631221224 **[Test build #122869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122869/testReport)** for PR 28586 at commit [`1079220`](https://github.com/apache/spark/commit/1079220509de19c889ce2a24683dc8e917a48bb1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631231179 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
SparkQA commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631230769 **[Test build #122869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122869/testReport)** for PR 28586 at commit [`1079220`](https://github.com/apache/spark/commit/1079220509de19c889ce2a24683dc8e917a48bb1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
AmplabJenkins commented on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631230092 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
AmplabJenkins removed a comment on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631230092 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
SparkQA commented on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631229769 **[Test build #122872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122872/testReport)** for PR 28589 at commit [`b4d215e`](https://github.com/apache/spark/commit/b4d215e9c4a8d6f336b6edc5ed6dce902c3debb5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
HyukjinKwon commented on pull request #28589: URL: https://github.com/apache/spark/pull/28589#issuecomment-631229613 This can be backported to branch-3.0 as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
gengliangwang commented on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631229677 @ajithme thanks for understanding! The commits are still in the master branch and they will be 3.1 as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #28589: [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace
HyukjinKwon opened a new pull request #28589: URL: https://github.com/apache/spark/pull/28589 ### What changes were proposed in this pull request? This PR proposes to only allow the import of `ResourceInformation` as below: ``` pyspark.resource.ResourceInformation ``` instead of ``` pyspark.ResourceInformation pyspark.resource.ResourceInformation ``` because `pyspark.resource` is a separate module, and it is documented so. The constructor of `ResourceInformation` isn't supposed to directly call anyway. ### Why are the changes needed? To keep the code structure coherent. ### Does this PR introduce _any_ user-facing change? No, it will be in the unreleased branches. ### How was this patch tested? Manually tested via importing: Before: ```python >>> import pyspark >>> pyspark.ResourceInformation >>> pyspark.resource.ResourceInformation ``` After: ```python >>> import pyspark >>> pyspark.ResourceInformation Traceback (most recent call last): File "", line 1, in AttributeError: module 'pyspark' has no attribute 'ResourceInformation' >>> pyspark.resource.ResourceInformation ``` Jenkins will test and existing tests should cover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme edited a comment on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
ajithme edited a comment on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631229046 agree. Its better to revert and we can reraise the PR on later release. will 3.x be a candidate for these PRs.? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
ajithme commented on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631229046 agree. Its better to revert and we can reraise the PR on later release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28575: [SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form
AmplabJenkins removed a comment on pull request #28575: URL: https://github.com/apache/spark/pull/28575#issuecomment-631228921 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28575: [SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form
AmplabJenkins commented on pull request #28575: URL: https://github.com/apache/spark/pull/28575#issuecomment-631228921 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
AmplabJenkins removed a comment on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631228387 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28575: [SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form
SparkQA removed a comment on pull request #28575: URL: https://github.com/apache/spark/pull/28575#issuecomment-631146069 **[Test build #122857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122857/testReport)** for PR 28575 at commit [`21fb7c5`](https://github.com/apache/spark/commit/21fb7c5844f43ec0b9190ac6e823aba2854ba2bd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
gengliangwang commented on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631228555 cc @erenavsarogullari @ajithme as well This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
AmplabJenkins commented on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631228387 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28575: [SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form
SparkQA commented on pull request #28575: URL: https://github.com/apache/spark/pull/28575#issuecomment-631228285 **[Test build #122857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122857/testReport)** for PR 28575 at commit [`21fb7c5`](https://github.com/apache/spark/commit/21fb7c5844f43ec0b9190ac6e823aba2854ba2bd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
SparkQA commented on pull request #28588: URL: https://github.com/apache/spark/pull/28588#issuecomment-631228091 **[Test build #122871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122871/testReport)** for PR 28588 at commit [`9e39968`](https://github.com/apache/spark/commit/9e39968d7eee987afc39cbd244ee4a2aef3dd4c6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #24076: [SPARK-27142][SQL] Provide REST API for SQL information
gengliangwang commented on pull request #24076: URL: https://github.com/apache/spark/pull/24076#issuecomment-631227143 @ajithme I think we have to revert this one in branch-3.0. See my comments in https://github.com/apache/spark/pull/28588 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28208: [SPARK-31440][SQL] Improve SQL Rest API
gengliangwang commented on pull request #28208: URL: https://github.com/apache/spark/pull/28208#issuecomment-631227040 @erenavsarogullari I think we have to revert this one in branch-3.0. See my comments in https://github.com/apache/spark/pull/28588 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang opened a new pull request #28588: Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0
gengliangwang opened a new pull request #28588: URL: https://github.com/apache/spark/pull/28588 ### What changes were proposed in this pull request? Revert https://github.com/apache/spark/pull/28208 and https://github.com/apache/spark/pull/24076 in branch 3.0 ### Why are the changes needed? Unfortunately, the PR https://github.com/apache/spark/pull/28208 is merged after Spark 3.0 RC 2 cut. Although the improvement is great, we can't break the policy to add new improvement commits into branch 3.0 now. Also, if we are going to adopt the improvement in a future release, we should not release 3.0 with https://github.com/apache/spark/pull/24076, since the API result will be changed. After discuss with @cloud-fan and @gatorsmile offline, we think the best choice is to revert both commits and follow community release policy. ### Does this PR introduce _any_ user-facing change? Yes, let's hold the SQL rest API until next release. ### How was this patch tested? Jenkins unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
AmplabJenkins removed a comment on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631225112 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
AmplabJenkins commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631225112 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
SparkQA removed a comment on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631221225 **[Test build #122868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122868/testReport)** for PR 28587 at commit [`26b8c9f`](https://github.com/apache/spark/commit/26b8c9f586bee29dc47cac5fec856eb9daa8601a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
SparkQA commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631225005 **[Test build #122868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122868/testReport)** for PR 28587 at commit [`26b8c9f`](https://github.com/apache/spark/commit/26b8c9f586bee29dc47cac5fec856eb9daa8601a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor
AmplabJenkins commented on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-631221498 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631221486 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631221486 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor
AmplabJenkins removed a comment on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-631221498 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
SparkQA commented on pull request #28587: URL: https://github.com/apache/spark/pull/28587#issuecomment-631221225 **[Test build #122868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122868/testReport)** for PR 28587 at commit [`26b8c9f`](https://github.com/apache/spark/commit/26b8c9f586bee29dc47cac5fec856eb9daa8601a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
SparkQA commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631221224 **[Test build #122869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122869/testReport)** for PR 28586 at commit [`1079220`](https://github.com/apache/spark/commit/1079220509de19c889ce2a24683dc8e917a48bb1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor
SparkQA commented on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-631221221 **[Test build #122870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122870/testReport)** for PR 26624 at commit [`d5c1aa9`](https://github.com/apache/spark/commit/d5c1aa97bd69a25de5de03ec79284f39dace3198). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor
cloud-fan commented on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-631220794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #28587: [SPARK-31766][K8S][TESTS] Add Spark version prefix to K8s UUID test image tag
dongjoon-hyun opened a new pull request #28587: URL: https://github.com/apache/spark/pull/28587 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
cloud-fan commented on a change in pull request #28576:
URL: https://github.com/apache/spark/pull/28576#discussion_r427726709
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##
@@ -31,17 +31,50 @@ import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy._
trait DateTimeFormatterHelper {
+ private def getOrDefault(accessor: TemporalAccessor, field: ChronoField,
default: Int): Int = {
+if (accessor.isSupported(field)) {
+ accessor.get(field)
+} else {
+ default
+}
+ }
+
+ protected def toLocalDate(accessor: TemporalAccessor, allowMissingYear:
Boolean): LocalDate = {
+val year = if (accessor.isSupported(ChronoField.YEAR)) {
+ accessor.get(ChronoField.YEAR)
+} else if (allowMissingYear) {
+ // To keep backward compatibility with Spark 2.x, we pick 1970 as the
default value of year.
+ 1970
+} else {
+ throw new SparkUpgradeException("3.0",
+"Year must be given in the date/timestamp string to be parsed. You can
set " +
+ SQLConf.LEGACY_ALLOW_MISSING_YEAR_DURING_PARSING.key + " to true, to
pick 1970 as " +
+ "the default value of year.", null)
+}
+val month = getOrDefault(accessor, ChronoField.MONTH_OF_YEAR, 1)
Review comment:
> asked for "mm" twice where they meant MM
This will error out unless the 2 mm parse to the same value, e.g. `mm mm
` with value `10 10 1990`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
cloud-fan commented on a change in pull request #28576:
URL: https://github.com/apache/spark/pull/28576#discussion_r427726478
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##
@@ -31,17 +31,50 @@ import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy._
trait DateTimeFormatterHelper {
+ private def getOrDefault(accessor: TemporalAccessor, field: ChronoField,
default: Int): Int = {
+if (accessor.isSupported(field)) {
+ accessor.get(field)
+} else {
+ default
+}
+ }
+
+ protected def toLocalDate(accessor: TemporalAccessor, allowMissingYear:
Boolean): LocalDate = {
+val year = if (accessor.isSupported(ChronoField.YEAR)) {
+ accessor.get(ChronoField.YEAR)
+} else if (allowMissingYear) {
+ // To keep backward compatibility with Spark 2.x, we pick 1970 as the
default value of year.
+ 1970
+} else {
+ throw new SparkUpgradeException("3.0",
+"Year must be given in the date/timestamp string to be parsed. You can
set " +
+ SQLConf.LEGACY_ALLOW_MISSING_YEAR_DURING_PARSING.key + " to true, to
pick 1970 as " +
+ "the default value of year.", null)
+}
+val month = getOrDefault(accessor, ChronoField.MONTH_OF_YEAR, 1)
Review comment:
January should be fine? It accepts any DAY_OF_MONTH.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
cloud-fan commented on a change in pull request #28576:
URL: https://github.com/apache/spark/pull/28576#discussion_r427726258
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##
@@ -31,17 +31,50 @@ import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy._
trait DateTimeFormatterHelper {
+ private def getOrDefault(accessor: TemporalAccessor, field: ChronoField,
default: Int): Int = {
+if (accessor.isSupported(field)) {
+ accessor.get(field)
+} else {
+ default
+}
+ }
+
+ protected def toLocalDate(accessor: TemporalAccessor, allowMissingYear:
Boolean): LocalDate = {
+val year = if (accessor.isSupported(ChronoField.YEAR)) {
+ accessor.get(ChronoField.YEAR)
+} else if (allowMissingYear) {
+ // To keep backward compatibility with Spark 2.x, we pick 1970 as the
default value of year.
+ 1970
+} else {
+ throw new SparkUpgradeException("3.0",
+"Year must be given in the date/timestamp string to be parsed. You can
set " +
Review comment:
A problem is that, the pattern is flexible and the year may not be in
the beginning, e.g. `MM/dd ...`. I don't think there is a workaround,
unless we provide an API to allow users to set the default values.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28523: [SPARK-31706][SQL] add back the support of streaming update mode
cloud-fan closed pull request #28523: URL: https://github.com/apache/spark/pull/28523 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28523: [SPARK-31706][SQL] add back the support of streaming update mode
cloud-fan commented on pull request #28523: URL: https://github.com/apache/spark/pull/28523#issuecomment-631218314 merging to master/3.0, thanks for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631217581 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122867/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
SparkQA removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631211570 **[Test build #122867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122867/testReport)** for PR 28586 at commit [`4615ed1`](https://github.com/apache/spark/commit/4615ed166d49aaf24b8c941b7faa8c4ccb7d25f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631217576 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631217576 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
SparkQA commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631217499 **[Test build #122867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122867/testReport)** for PR 28586 at commit [`4615ed1`](https://github.com/apache/spark/commit/4615ed166d49aaf24b8c941b7faa8c4ccb7d25f0). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins removed a comment on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631211891 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
AmplabJenkins commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631211891 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28574: [SPARK-31752][SQL][DOCS] Add sql doc for interval type
yaooqinn commented on pull request #28574: URL: https://github.com/apache/spark/pull/28574#issuecomment-631211798 One thing is how to categorize all data types, and the other is to decide the outline and content based on the generality and difference of those types. We may set a good example here and improve it whenever needed. https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html https://www.postgresql.org/docs/9.0/datatype.html https://dev.mysql.com/doc/refman/8.0/en/data-types.html https://docs.snowflake.com/en/sql-reference/data-types.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
SparkQA commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631211570 **[Test build #122867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122867/testReport)** for PR 28586 at commit [`4615ed1`](https://github.com/apache/spark/commit/4615ed166d49aaf24b8c941b7faa8c4ccb7d25f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
HyukjinKwon commented on pull request #28586: URL: https://github.com/apache/spark/pull/28586#issuecomment-631210515 cc @dongjoon-hyun, can you take a quick look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #28586: [SPARK-31748][PYTHON][3.0] Document resource module in PySpark doc and rename/move classes
HyukjinKwon opened a new pull request #28586: URL: https://github.com/apache/spark/pull/28586 ### What changes were proposed in this pull request? This PR partially backports https://github.com/apache/spark/pull/28569. 1.. Rename ``` pyspark └── resourceinformation.py └── class ResourceInformation ``` to ``` pyspark └── resource.py └── class ResourceInformation ``` So, the `ResourceInformation` is consistently imported via `pyspark.resource.ResourceInformation`. 2.. Document the new `pyspark.resource` module 3.. Minor docstring fix e.g.: ```diff - @param name the name of the resource - @param addresses an array of strings describing the addresses of the resource + :param name: the name of the resource + :param addresses: an array of strings describing the addresses of the resource + + .. versionadded:: 3.0.0 ``` ### Why are the changes needed? To document APIs, and move Python modules to fewer and simpler modules. ### Does this PR introduce _any_ user-facing change? No, the changes are in unreleased branches. ### How was this patch tested? Manually tested via: ```bash cd python ./run-tests --python-executables=python3 --modules=pyspark-core ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
AmplabJenkins removed a comment on pull request #28578: URL: https://github.com/apache/spark/pull/28578#issuecomment-631209414 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
AmplabJenkins commented on pull request #28578: URL: https://github.com/apache/spark/pull/28578#issuecomment-631209414 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
SparkQA removed a comment on pull request #28578: URL: https://github.com/apache/spark/pull/28578#issuecomment-631154272 **[Test build #122863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122863/testReport)** for PR 28578 at commit [`6ccbc39`](https://github.com/apache/spark/commit/6ccbc391067995205127425ef59c4a9f1fe40520). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
SparkQA commented on pull request #28578: URL: https://github.com/apache/spark/pull/28578#issuecomment-631208800 **[Test build #122863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122863/testReport)** for PR 28578 at commit [`6ccbc39`](https://github.com/apache/spark/commit/6ccbc391067995205127425ef59c4a9f1fe40520). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
sarutak commented on a change in pull request #28585:
URL: https://github.com/apache/spark/pull/28585#discussion_r427716093
##
File path: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
##
@@ -500,7 +501,11 @@ private[spark] case class ServerInfo(
threadPool match {
case pool: QueuedThreadPool =>
// Workaround for SPARK-30385 to avoid Jetty's acceptor thread shrink.
-pool.setIdleTimeout(0)
+// As of Jetty 9.4.21, the implementation of
+// QueuedThreadPoolExecutor#setIdleTimeout is changed and
IllegalStateException
+// will be thrown if we try to set idle timeout after the server has
started.
+// But this workaround works for Jetty 9.4.28 by ignoring the
exception.
+Try(pool.setIdleTimeout(0))
Review comment:
I noticed that as of Jetty 9.4.21, the implementation of
`QueuedThreadPool`.
When `QueuedThreadPool#setIdleTimeout` is called,
`QueuedThreadPool#_idleTimeout` is set to the new timeout as well as
[`ReservedThreadExecutor#_idleTimeout`](https://github.com/eclipse/jetty.project/blob/jetty-9.4.28.v20200408/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L312).
If `ReservedThreadExecutor#_idleTImeout` is tried to be set after Jetty has
started, [`IllegalStateException` will be
thrown](https://github.com/eclipse/jetty.project/blob/jetty-9.4.28.v20200408/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ReservedThreadExecutor.java#L149).
But `QueuedThreadPool#_idleTimeout` will be set so this workaround will
still work by ignoring the exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631206283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631206283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
SparkQA commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-631205985 **[Test build #122866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122866/testReport)** for PR 28585 at commit [`f53735e`](https://github.com/apache/spark/commit/f53735e2e52a8cf3d70f162c3317c10383ae1aff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak opened a new pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
sarutak opened a new pull request #28585: URL: https://github.com/apache/spark/pull/28585 ### What changes were proposed in this pull request? This PR upgrades HtmlUnit. Selenium and Jetty also upgraded because of dependency. ### Why are the changes needed? Recently, a security issue which affects HtmlUnit is reported. https://nvd.nist.gov/vuln/detail/CVE-2020-5529 According to the report, arbitrary code can be run by malicious users. HtmlUnit is used for test so the impact might not be large but it's better to upgrade it just in case. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing testcases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28580: [SPARK-31759][Deploy] Support configurable max number of rotate logs for spark daemons
maropu commented on a change in pull request #28580:
URL: https://github.com/apache/spark/pull/28580#discussion_r427710973
##
File path: sbin/spark-daemon.sh
##
@@ -74,10 +75,13 @@ shift
spark_rotate_log ()
{
log=$1;
-num=5;
-if [ -n "$2" ]; then
- num=$2
+
+if [[ ${SPARK_LOG_MAX_FILES} -gt 5 ]] 2>/dev/null; then
+ num=${SPARK_LOG_MAX_FILES}
+else
+ num=5
Review comment:
Oh, I see. I got it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28511: [SPARK-31684][SQL] Overwrite partition failed with 'WRONG FS' when the target partition is not belong to the filesystem as same as the table
yaooqinn commented on pull request #28511: URL: https://github.com/apache/spark/pull/28511#issuecomment-631198476 thanks for reviewing and merging! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
AmplabJenkins removed a comment on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631197828 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28584: [SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite
AmplabJenkins removed a comment on pull request #28584: URL: https://github.com/apache/spark/pull/28584#issuecomment-631197375 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122862/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
