[GitHub] [spark] HyukjinKwon commented on issue #28122: Add .asf.yaml to control Github settings
HyukjinKwon commented on issue #28122: Add .asf.yaml to control Github settings URL: https://github.com/apache/spark/pull/28122#issuecomment-609363823 I think it seems fine to have. Mind filing a JIRA please? See also https://spark.apache.org/contributing.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28119: [WIP][SQL] Speed up timestamps rebasing
AmplabJenkins removed a comment on issue #28119: [WIP][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-609363632 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28119: [WIP][SQL] Speed up timestamps rebasing
AmplabJenkins commented on issue #28119: [WIP][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-609363634 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120824/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28119: [WIP][SQL] Speed up timestamps rebasing
AmplabJenkins commented on issue #28119: [WIP][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-609363632 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28119: [WIP][SQL] Speed up timestamps rebasing
AmplabJenkins removed a comment on issue #28119: [WIP][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-609363634 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120824/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609363510 **[Test build #120828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120828/testReport)** for PR 28125 at commit [`98fe6dc`](https://github.com/apache/spark/commit/98fe6dc2109f7fb49b6632026fea561ee81b3935). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609363537 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120828/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609363533 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609363537 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120828/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28119: [WIP][SQL] Speed up timestamps rebasing
SparkQA commented on issue #28119: [WIP][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-609363508 **[Test build #120824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120824/testReport)** for PR 28119 at commit [`ac2beef`](https://github.com/apache/spark/commit/ac2beefe95e8a2a2c583355b10ee4ba0820f8021). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28119: [WIP][SQL] Speed up timestamps rebasing
SparkQA removed a comment on issue #28119: [WIP][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-609114842 **[Test build #120824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120824/testReport)** for PR 28119 at commit [`ac2beef`](https://github.com/apache/spark/commit/ac2beefe95e8a2a2c583355b10ee4ba0820f8021). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609362700 **[Test build #120828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120828/testReport)** for PR 28125 at commit [`98fe6dc`](https://github.com/apache/spark/commit/98fe6dc2109f7fb49b6632026fea561ee81b3935). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609363533 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609362826 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25527/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609362823 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609362823 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609362826 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25527/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609362700 **[Test build #120828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120828/testReport)** for PR 28125 at commit [`98fe6dc`](https://github.com/apache/spark/commit/98fe6dc2109f7fb49b6632026fea561ee81b3935). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
HyukjinKwon commented on a change in pull request #28125: [SPARK-31351][DOC]
Migration Guide Auditing for Spark 3.0 Release
URL: https://github.com/apache/spark/pull/28125#discussion_r403651993
##
File path: docs/css/main.css
##
@@ -2,6 +2,37 @@
Author's custom styles
==
*/
+table {
Review comment:
Ah, nice. +1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r403651608
##
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##
@@ -897,6 +901,18 @@ public void persist() {
assertEquals(1, rdd.first().intValue());
}
+ @Test
Review comment:
@tgravescs, what about disabling this test and see if it was from side
effect? Logically, Python related logic does not affect Scala/Java test cases,
and I presume it might be the side effect from the test it runs first.
Seems the test output is from somewhere related to matrix (e.g.,
`MetricsSystemSuite`); however, I am not sure where exactly it comes from.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27593: [SPARK-30818][SPARKR][ML] Add SparkR LinearRegression wrapper
huaxingao commented on a change in pull request #27593:
[SPARK-30818][SPARKR][ML] Add SparkR LinearRegression wrapper
URL: https://github.com/apache/spark/pull/27593#discussion_r403651225
##
File path: R/pkg/R/mllib_regression.R
##
@@ -540,3 +546,149 @@ setMethod("write.ml", signature(object =
"AFTSurvivalRegressionModel", path = "c
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' Linear Regression Model
+#'
+#' \code{spark.lm} fits a linear regression model against a SparkDataFrame.
+#' Users can call \code{summary} to print a summary of the fitted model,
+#' \code{predict} to make predictions on new data,
+#' and \code{write.ml}/\code{read.ml} to save/load fitted models.
+#'
+#' @param data a \code{SparkDataFrame} of observations and labels for model
fitting.
+#' @param formula a symbolic description of the model to be fitted. Currently
only a few formula
+#'operators are supported, including '~', '.', ':', '+', and
'-'.
+#' @param maxIter maximum iteration number.
+#' @param regParam the regularization parameter.
+#' @param elasticNetParam the ElasticNet mixing parameter, in range [0, 1].
+#'For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an
L1 penalty.
+#' @param tol convergence tolerance of iterations.
+#' @param standardization whether to standardize the training features before
fitting the model.
+#' @param weightCol weight column name.
+#' @param aggregationDepth suggested depth for treeAggregate (>= 2).
+#' @param loss the loss function to be optimized. Supported options:
"squaredError" and "huber".
+#' @param epsilon the shape parameter to control the amount of robustness.
+#' @param solver The solver algorithm for optimization.
+#'Supported options: "l-bfgs", "normal" and "auto".
+#' @param stringIndexerOrderType how to order categories of a string feature
column. This is used to
+#' decide the base level of a string feature as
the last category
+#' after ordering is dropped when encoding
strings. Supported options
+#' are "frequencyDesc", "frequencyAsc",
"alphabetDesc", and
+#' "alphabetAsc". The default value is
"frequencyDesc". When the
+#' ordering is set to "alphabetDesc", this drops
the same category
+#' as R when encoding strings.
+#' @param ... additional arguments passed to the method.
+#' @return \code{spark.lm} returns a fitted Linear Regression Model.
+#'
Review comment:
@HyukjinKwon Is this blank line OK? I remember in one of my PR, Felix said
the blank line is significant in roxygen2 and asked me to remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403650987 ## File path: python/pyspark/__init__.py ## @@ -55,6 +55,12 @@ from pyspark.accumulators import Accumulator, AccumulatorParam from pyspark.broadcast import Broadcast from pyspark.resourceinformation import ResourceInformation +from pyspark.taskresourcerequest import TaskResourceRequest +from pyspark.executorresourcerequest import ExecutorResourceRequest +from pyspark.taskresourcerequests import TaskResourceRequests +from pyspark.executorresourcerequests import ExecutorResourceRequests +from pyspark.resourceprofilebuilder import ResourceProfileBuilder +from pyspark.resourceprofile import ResourceProfile Review comment: Can we group these into less modules? e.g., - `resourceprofile` and `resourceprofilebuilder` -> `resource_profile` - `executorresourcerequest` and `executorresourcerequests` -> `resource_request`, - `taskresourcerequest` and `taskresourcerequests` -> `resource_request` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable()
AmplabJenkins removed a comment on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable() URL: https://github.com/apache/spark/pull/27908#issuecomment-609360284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609360256 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609360258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120827/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable()
AmplabJenkins commented on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable() URL: https://github.com/apache/spark/pull/27908#issuecomment-609360286 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120823/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable()
AmplabJenkins commented on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable() URL: https://github.com/apache/spark/pull/27908#issuecomment-609360284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable()
AmplabJenkins removed a comment on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable() URL: https://github.com/apache/spark/pull/27908#issuecomment-609360286 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120823/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609360258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120827/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609360256 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359542 **[Test build #120827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120827/testReport)** for PR 28125 at commit [`e24fe8e`](https://github.com/apache/spark/commit/e24fe8e3b1a330433b566b611cb6073608508e41). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609360235 **[Test build #120827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120827/testReport)** for PR 28125 at commit [`e24fe8e`](https://github.com/apache/spark/commit/e24fe8e3b1a330433b566b611cb6073608508e41). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable()
SparkQA removed a comment on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable() URL: https://github.com/apache/spark/pull/27908#issuecomment-609110876 **[Test build #120823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120823/testReport)** for PR 27908 at commit [`8c90a93`](https://github.com/apache/spark/commit/8c90a9319ebfb7235da514fede96dc74ae2906c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable()
SparkQA commented on issue #27908: [SPARK-31000] Add ability to set table description via Catalog.createTable() URL: https://github.com/apache/spark/pull/27908#issuecomment-609360150 **[Test build #120823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120823/testReport)** for PR 27908 at commit [`8c90a93`](https://github.com/apache/spark/commit/8c90a9319ebfb7235da514fede96dc74ae2906c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120826/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359927 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359903 **[Test build #120826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120826/testReport)** for PR 28125 at commit [`506fcfb`](https://github.com/apache/spark/commit/506fcfbcf3296389fc3701f9d8b46fde611be692). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609350837 **[Test build #120826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120826/testReport)** for PR 28125 at commit [`506fcfb`](https://github.com/apache/spark/commit/506fcfbcf3296389fc3701f9d8b46fde611be692). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359927 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120826/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r403649296
##
File path: python/pyspark/executorresourcerequest.py
##
@@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+class ExecutorResourceRequest(object):
+"""
+.. note:: Evolving
+
+An Executor resource request. This is used in conjunction with the
ResourceProfile to
+programmatically specify the resources needed for an RDD that will be
applied at the
+stage level.
+
+This is used to specify what the resource requirements are for an Executor
and how
+Spark can find out specific details about those resources. Not all the
parameters are
+required for every resource type. Resources like GPUs are supported and
have same limitations
+as using the global spark configs spark.executor.resource.gpu.*. The
amount, discoveryScript,
+and vendor parameters for resources are all the same parameters a user
would specify through the
+configs: spark.executor.resource.{resourceName}.{amount, discoveryScript,
vendor}.
+
+For instance, a user wants to allocate an Executor with GPU resources on
YARN. The user has
+to specify the resource name (gpu), the amount or number of GPUs per
Executor,
+the discovery script would be specified so that when the Executor starts
up it can
+discovery what GPU addresses are available for it to use because YARN
doesn't tell
+Spark that, then vendor would not be used because its specific for
Kubernetes.
+
+See the configuration and cluster specific docs for more details.
+
+Use ExecutorResourceRequests class as a convenience API.
Review comment:
I would link it properly by, for example, using ``
:class:`pyspark.ExecutorResourceRequests` ``
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403649190 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1135,6 +1137,22 @@ private[spark] class DAGScheduler( } } + /** + * PythonRunner needs to know what the pyspark memory setting is for the profile being run. Review comment: nit: PythonRunner -> `` `PythonRunner` ``or `[[PythonRunner]]`` since it's a `class` which doesn't have any problem in unidoc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25526/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25526/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359624 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359624 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403649190 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1135,6 +1137,22 @@ private[spark] class DAGScheduler( } } + /** + * PythonRunner needs to know what the pyspark memory setting is for the profile being run. Review comment: nit: PythonRunner -> `` `PythonRunner` ``. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403649190 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1135,6 +1137,22 @@ private[spark] class DAGScheduler( } } + /** + * PythonRunner needs to know what the pyspark memory setting is for the profile being run. Review comment: nit: PythonRunner -> `` `PythonRunner` ``or `[[PythonRunner]]`` since it's a `class` which doesn't have any problem in unidoc.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609359542 **[Test build #120827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120827/testReport)** for PR 28125 at commit [`e24fe8e`](https://github.com/apache/spark/commit/e24fe8e3b1a330433b566b611cb6073608508e41). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403648909 ## File path: python/pyspark/__init__.py ## @@ -120,4 +126,6 @@ def wrapper(self, *args, **kwargs): "Accumulator", "AccumulatorParam", "MarshalSerializer", "PickleSerializer", "StatusTracker", "SparkJobInfo", "SparkStageInfo", "Profiler", "BasicProfiler", "TaskContext", "RDDBarrier", "BarrierTaskContext", "BarrierTaskInfo", "ResourceInformation", +"TaskResourceRequest", "TaskResourceRequests", "ExecutorResourceRequest", +"ExecutorResourceRequests", "ResourceProfile" Review comment: @tgravescs, please don't add non-API instances here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609352900 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25525/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins removed a comment on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609352873 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609352900 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25525/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
gatorsmile commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609352683 cc @cloud-fan @srowen @HyukjinKwon @dongjoon-hyun @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
AmplabJenkins commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609352873 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27570: [SPARK-30820][SPARKR][ML] Add FMClassifier to SparkR
huaxingao commented on a change in pull request #27570:
[SPARK-30820][SPARKR][ML] Add FMClassifier to SparkR
URL: https://github.com/apache/spark/pull/27570#discussion_r403648627
##
File path: R/pkg/tests/fulltests/test_mllib_classification.R
##
@@ -488,4 +488,36 @@ test_that("spark.naiveBayes", {
expect_equal(class(collect(predictions)$clicked[1]), "character")
})
+test_that("spark.fmClassifier", {
+ df <- withColumn(
+suppressWarnings(createDataFrame(iris)),
+"Species", otherwise(when(column("Species") == "Setosa", "Setosa"),
"Not-Setosa")
+ )
+
+ model1 <- spark.fmClassifier(
+df, Species ~ .,
+regParam = 0.01, maxIter = 10, fitLinear = TRUE, factorSize = 3
+ )
+
+ prediction1 <- predict(model1, df)
+ expect_is(prediction1, "SparkDataFrame")
Review comment:
I am OK with this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
SparkQA commented on issue #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125#issuecomment-609350837 **[Test build #120826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120826/testReport)** for PR 28125 at commit [`506fcfb`](https://github.com/apache/spark/commit/506fcfbcf3296389fc3701f9d8b46fde611be692). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403648582 ## File path: python/pyspark/executorresourcerequest.py ## @@ -0,0 +1,73 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +class ExecutorResourceRequest(object): +""" +.. note:: Evolving Review comment: Seems it's not an API in Scala side. Seems better to remove this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile opened a new pull request #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release
gatorsmile opened a new pull request #28125: [SPARK-31351][DOC] Migration Guide Auditing for Spark 3.0 Release URL: https://github.com/apache/spark/pull/28125 ### What changes were proposed in this pull request? This PR is to audit the migration guides in Spark 3.0 release: - correct the grammar errors - replace HTML table by markdown table ### Why are the changes needed? N/A ### Does this PR introduce any user-facing change? No ### How was this patch tested? Screenshot:     This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403648497 ## File path: core/src/main/scala/org/apache/spark/api/java/JavaRDD.scala ## @@ -49,6 +50,20 @@ class JavaRDD[T](val rdd: RDD[T])(implicit val classTag: ClassTag[T]) */ def persist(newLevel: StorageLevel): JavaRDD[T] = wrapRDD(rdd.persist(newLevel)) + /** + * Specify a ResourceProfile to use when calculating this RDD. This is only supported on + * certain cluster managers and currently requires dynamic allocation to be enabled. + * It will result in new executors with the resources specified being acquired to + * calculate the RDD. + */ + def withResources(prof: ResourceProfile): JavaRDD[T] = wrapRDD(rdd.withResources(prof)) Review comment: Please use `@Private` annotation if this is meant to be private yet: https://github.com/apache/spark/blob/9610efc252c94f93689d45e320df1c5815d97b25/common/tags/src/main/java/org/apache/spark/annotation/Private.java#L26-L28 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
HyukjinKwon commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r403648232 ## File path: python/pyspark/rdd.py ## @@ -2483,6 +2485,32 @@ def _is_barrier(self): """ return self._jrdd.rdd().isBarrier() +def withResources(self, profile): +""" +.. note:: Experimental + +Specify a ResourceProfile to use when calculating this RDD. This is only supported on +certain cluster managers and currently requires dynamic allocation to be enabled. +It will result in new executors with the resources specified being acquired to +calculate the RDD. + +.. versionadded:: 3.1.0 +""" +self.has_resourceProfile = True +self._jrdd.withResources(profile._jResourceProfile) +return self + +def getResourceProfile(self): Review comment: @tgravescs, can you clarify if we're going to make `getResourceProfile` and `withResources` as APIs or not first? I strongly think this is a bad idea to mix private and API declarations https://github.com/apache/spark/blob/5d76b12e9b2ca0eb090c3c5145eee4cf78caba13/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1740-L1743 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403647618
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
+ Column name
+ Returns the average of values in the input column.
+
+
+ {bool_and | every}(e: Column)
+ Column name
+ Returns true if all values are true
+
+
+ {any | some | bool_or}(e: Column)
+ Column name
+ Returns true if at least one value is true
+
+
+ approx_count_distinct(e: Column)
+ Column name
+ Returns the estimated cardinality by HyperLogLog++
+
+
+ corr(e1: Column, e2: Column)
+ Column name
+ Returns Pearson coefficient of correlation between a set of number
pairs
+
+
+ count(*)
+ None
+ Returns the total number of retrieved rows, including rows
containing null
+
+
+ count(e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are all
not null
+
+
+ count(DISTINCT e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are
unique and not null
+
+
+ count_if(Predicate)
+ Expression that will be used for aggregation calculation
+ Returns the count number from the predicate evaluate to `TRUE`
values
Review comment:
backtick doesn't work inside html, use ```TRUE```?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403647671
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
+ Column name
+ Returns the average of values in the input column.
+
+
+ {bool_and | every}(e: Column)
+ Column name
+ Returns true if all values are true
+
+
+ {any | some | bool_or}(e: Column)
+ Column name
+ Returns true if at least one value is true
+
+
+ approx_count_distinct(e: Column)
+ Column name
+ Returns the estimated cardinality by HyperLogLog++
+
+
+ corr(e1: Column, e2: Column)
+ Column name
+ Returns Pearson coefficient of correlation between a set of number
pairs
+
+
+ count(*)
+ None
+ Returns the total number of retrieved rows, including rows
containing null
+
+
+ count(e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are all
not null
+
+
+ count(DISTINCT e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are
unique and not null
+
+
+ count_if(Predicate)
+ Expression that will be used for aggregation calculation
+ Returns the count number from the predicate evaluate to `TRUE`
values
+
+
+ covar_pop(e1: Column, e2: Column)
+ Column name
+ Returns the population covariance of a set of number pairs
+
+
+ covar_samp(e1: Column, e2: Column)
+ Column name
+ Returns the sample covariance of a set of number pairs
+
+
+ {first | first_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the first value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
+
+
+skewness(e: Column)
+ Column name
+ Returns the skewness value calculated from values of a group
+
+
+kurtosis(e: Column)
+ Column name
+ Returns the kurtosis value calculated from values of a group
+
+
+ {last | last_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the last value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
+
+
+ max(e: Column)
+ Column name
+ Returns the maximum value of the column.
+
+
+ max_by(e1: Column, e2: Column)
+ Column name
+ Returns the value of column e1 associated with the maximum value of
column e2.
+
+
+ min(e: Column)
+ Column name
+ Returns the minimum value of the column.
+
+
+ min_by(e1: Column, e2: Column)
+ Column name
+ Returns the value of column e1 associated with the minimum value of
column e2.
+
+
+ percentile(e: Column, percentage [, frequency])
+ Column name; percentage is a number between 0 and 1; frequency is a
positive integer
+ Returns the exact percentile value of numeric column at the given
percentage.
+
+
+ percentile(e: Column, array(percentage1 [,
percentage2]...) [, frequency])
+ Column name; percentage array is an array of number between 0 and 1;
frequency is a positive integer
+ Returns the exact percentile value array of numeric column at the
given percentage(s).
+
+
+ {percentile_approx | percentile_approx}(e: Column,
percentage [, frequency])
Review comment:
Is this a 3.1 function?
```
* @group agg_funcs
* @since 3.1.0
*/
def percentile_approx(e: Column, percentage: Column, accuracy: Column):
Column = {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403647990
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
+ Column name
+ Returns the average of values in the input column.
+
+
+ {bool_and | every}(e: Column)
+ Column name
+ Returns true if all values are true
+
+
+ {any | some | bool_or}(e: Column)
+ Column name
+ Returns true if at least one value is true
+
+
+ approx_count_distinct(e: Column)
+ Column name
+ Returns the estimated cardinality by HyperLogLog++
+
+
+ corr(e1: Column, e2: Column)
+ Column name
+ Returns Pearson coefficient of correlation between a set of number
pairs
+
+
+ count(*)
+ None
+ Returns the total number of retrieved rows, including rows
containing null
+
+
+ count(e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are all
not null
+
+
+ count(DISTINCT e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are
unique and not null
+
+
+ count_if(Predicate)
+ Expression that will be used for aggregation calculation
+ Returns the count number from the predicate evaluate to `TRUE`
values
+
+
+ covar_pop(e1: Column, e2: Column)
+ Column name
+ Returns the population covariance of a set of number pairs
+
+
+ covar_samp(e1: Column, e2: Column)
+ Column name
+ Returns the sample covariance of a set of number pairs
+
+
+ {first | first_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the first value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
+
+
+skewness(e: Column)
+ Column name
+ Returns the skewness value calculated from values of a group
+
+
+kurtosis(e: Column)
+ Column name
+ Returns the kurtosis value calculated from values of a group
+
+
+ {last | last_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the last value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
Review comment:
```isIgnoreNull```?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403647588
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
+ Column name
+ Returns the average of values in the input column.
+
+
+ {bool_and | every}(e: Column)
+ Column name
+ Returns true if all values are true
+
+
+ {any | some | bool_or}(e: Column)
+ Column name
+ Returns true if at least one value is true
+
+
+ approx_count_distinct(e: Column)
Review comment:
It has the optional ```relativeSD```. Change to
```approx_count_distinct(expr[, relativeSD])```?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403647782
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
+ Column name
+ Returns the average of values in the input column.
+
+
+ {bool_and | every}(e: Column)
+ Column name
+ Returns true if all values are true
+
+
+ {any | some | bool_or}(e: Column)
+ Column name
+ Returns true if at least one value is true
+
+
+ approx_count_distinct(e: Column)
+ Column name
+ Returns the estimated cardinality by HyperLogLog++
+
+
+ corr(e1: Column, e2: Column)
+ Column name
+ Returns Pearson coefficient of correlation between a set of number
pairs
+
+
+ count(*)
+ None
+ Returns the total number of retrieved rows, including rows
containing null
+
+
+ count(e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are all
not null
+
+
+ count(DISTINCT e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are
unique and not null
+
+
+ count_if(Predicate)
+ Expression that will be used for aggregation calculation
+ Returns the count number from the predicate evaluate to `TRUE`
values
+
+
+ covar_pop(e1: Column, e2: Column)
+ Column name
+ Returns the population covariance of a set of number pairs
+
+
+ covar_samp(e1: Column, e2: Column)
+ Column name
+ Returns the sample covariance of a set of number pairs
+
+
+ {first | first_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the first value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
+
+
+skewness(e: Column)
+ Column name
+ Returns the skewness value calculated from values of a group
+
+
+kurtosis(e: Column)
+ Column name
+ Returns the kurtosis value calculated from values of a group
+
+
+ {last | last_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the last value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
+
+
+ max(e: Column)
+ Column name
+ Returns the maximum value of the column.
+
+
+ max_by(e1: Column, e2: Column)
+ Column name
+ Returns the value of column e1 associated with the maximum value of
column e2.
+
+
+ min(e: Column)
+ Column name
+ Returns the minimum value of the column.
+
+
+ min_by(e1: Column, e2: Column)
+ Column name
+ Returns the value of column e1 associated with the minimum value of
column e2.
+
+
+ percentile(e: Column, percentage [, frequency])
+ Column name; percentage is a number between 0 and 1; frequency is a
positive integer
+ Returns the exact percentile value of numeric column at the given
percentage.
+
+
+ percentile(e: Column, array(percentage1 [,
percentage2]...) [, frequency])
+ Column name; percentage array is an array of number between 0 and 1;
frequency is a positive integer
+ Returns the exact percentile value array of numeric column at the
given percentage(s).
+
+
+ {percentile_approx | percentile_approx}(e: Column,
percentage [, frequency])
+ Column name; percentage is a number between 0 and 1; frequency is a
positive integer
+ Returns the approximate percentile value of numeric column at the
given percentage.
+
+
+ {percentile_approx | percentile_approx}(e: Column,
array(percentage1 [, percentage2]...) [, frequency])
+ Column name; percentage is a number between 0 and 1; frequency is a
positive integer
+ Returns the approximate percentile value of numeric column at the
given percentage.
+
+
+ {stddev_samp |
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate URL: https://github.com/apache/spark/pull/28120#discussion_r403644369 ## File path: docs/sql-ref-functions-builtin-aggregate.md ## @@ -19,4 +19,626 @@ license: | limitations under the License. --- -Aggregate functions \ No newline at end of file +Spark SQL provides build-in Aggregate functions defines in dataset API and SQL interface. Aggregate functions Review comment: ```defines in dataset API``` -> ```defined in the dataset API```? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate URL: https://github.com/apache/spark/pull/28120#discussion_r403647912 ## File path: docs/sql-ref-functions-builtin-aggregate.md ## @@ -19,4 +19,626 @@ license: | limitations under the License. --- -Aggregate functions \ No newline at end of file +Spark SQL provides build-in Aggregate functions defines in dataset API and SQL interface. Aggregate functions +operate on a group of rows and return a single value. + +Spark SQL Aggregate functions are grouped as agg_funcs in spark SQL. Below is the list of functions. + +**Note:** Every below function has another signature which take String as a column name instead of Column. Review comment: ```which take String``` -> ```which takes String```? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403644510
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
Review comment:
Could you list the functions in alphabetical order?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28120: [SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
huaxingao commented on a change in pull request #28120:
[SPARK-31349][SQL][DOCS] Sql ref buildin-aggregate
URL: https://github.com/apache/spark/pull/28120#discussion_r403647624
##
File path: docs/sql-ref-functions-builtin-aggregate.md
##
@@ -19,4 +19,626 @@ license: |
limitations under the License.
---
-Aggregate functions
\ No newline at end of file
+Spark SQL provides build-in Aggregate functions defines in dataset API and SQL
interface. Aggregate functions
+operate on a group of rows and return a single value.
+
+Spark SQL Aggregate functions are grouped as agg_funcs in spark
SQL. Below is the list of functions.
+
+**Note:** Every below function has another signature which take String as a
column name instead of Column.
+
+* Table of contents
+{:toc}
+
+
+FunctionParametersDescription
+
+
+
+ {avg | mean}(e: Column)
+ Column name
+ Returns the average of values in the input column.
+
+
+ {bool_and | every}(e: Column)
+ Column name
+ Returns true if all values are true
+
+
+ {any | some | bool_or}(e: Column)
+ Column name
+ Returns true if at least one value is true
+
+
+ approx_count_distinct(e: Column)
+ Column name
+ Returns the estimated cardinality by HyperLogLog++
+
+
+ corr(e1: Column, e2: Column)
+ Column name
+ Returns Pearson coefficient of correlation between a set of number
pairs
+
+
+ count(*)
+ None
+ Returns the total number of retrieved rows, including rows
containing null
+
+
+ count(e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are all
not null
+
+
+ count(DISTINCT e: Column[, e: Column])
+ Column name
+ Returns the number of rows for which the supplied column(s) are
unique and not null
+
+
+ count_if(Predicate)
+ Expression that will be used for aggregation calculation
+ Returns the count number from the predicate evaluate to `TRUE`
values
+
+
+ covar_pop(e1: Column, e2: Column)
+ Column name
+ Returns the population covariance of a set of number pairs
+
+
+ covar_samp(e1: Column, e2: Column)
+ Column name
+ Returns the sample covariance of a set of number pairs
+
+
+ {first | first_value}(e: Column[, isIgnoreNull])
+ Column name[, True/False(default)]
+ Returns the first value of column for a group of rows. If
`isIgnoreNull` is true, returns only non-null values, default is false. This
function is non-deterministic
Review comment:
```isIgnoreNull```?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-
HyukjinKwon edited a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609312136 I picked/verified each commit related to this PR manually to branch-3.0. @beliefer, do you mean keep the commit logs in your PR so that I can manually cherry-pick the commits from your PR to branch-3.0? Also, it would be nice if we mention the difference against the master. For example, I found three differences: https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-8b4ea8f3b0cc1e7ce7e943de1abbb165R66 https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-8b4ea8f3b0cc1e7ce7e943de1abbb165R72 https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-b2fc8d6ab7ac5735085e2d6cfacb95daR88 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-
HyukjinKwon edited a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609312136 I picked/verified each commit related to this PR manually to branch-3.0. @beliefer, do you mean keep the commit logs in your PR so that I can manually cherry-pick the commits from your PR to branch-3.0? Also, it would be nice if we mention the difference against the master. For example, I found three differences: https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-8b4ea8f3b0cc1e7ce7e943de1abbb165R66 https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-8b4ea8f3b0cc1e7ce7e943de1abbb165R72 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPA
AmplabJenkins removed a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609315334 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120825/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPA
AmplabJenkins removed a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609315285 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914
AmplabJenkins commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609315285 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28112: [SPARK-31315][SQL][FOLLOWUP][MINOR] Fix some typo and improve comments.
HyukjinKwon commented on issue #28112: [SPARK-31315][SQL][FOLLOWUP][MINOR] Fix some typo and improve comments. URL: https://github.com/apache/spark/pull/28112#issuecomment-609315479 cc @rednaxelafx since the comment is from you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914
AmplabJenkins commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609315334 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120825/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-309
SparkQA removed a comment on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609116983 **[Test build #120825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120825/testReport)** for PR 28124 at commit [`9fc3624`](https://github.com/apache/spark/commit/9fc3624ddea3699e193b66b3d6eb1477d5309970). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0]
SparkQA commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609312219 **[Test build #120825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120825/testReport)** for PR 28124 at commit [`9fc3624`](https://github.com/apache/spark/commit/9fc3624ddea3699e193b66b3d6eb1477d5309970). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914]
HyukjinKwon closed pull request #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][
HyukjinKwon commented on issue #28124: [SPARK-30887][SPARK-30888][SPARK-30889][SPARK-30891][SPARK-30908][SPARK-30909][SPARK-30910][SPARK-30911][SPARK-30912][SPARK-30913][SPARK-30914][3.0] Add version information to the configuration of Deploy, History, NetWork, Worker, Tests, R, Python, Status, Kryo, UI URL: https://github.com/apache/spark/pull/28124#issuecomment-609312136 I picked/verified each commit related to this PR manually to branch-3.0. @beliefer, do you mean keep the commit logs in your PR so that I can manually cherry-pick the commits from your PR to branch-3.0? Also, it would be nice if we mention the difference against the master. For example, I found two differences: https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-8b4ea8f3b0cc1e7ce7e943de1abbb165R66 https://github.com/apache/spark/commit/ebcff675e0c8c915075541af8fa8f9a24087ac11#diff-8b4ea8f3b0cc1e7ce7e943de1abbb165R72 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28118: [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core
dongjoon-hyun commented on issue #28118: [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/28118#issuecomment-609309817 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28118: [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core
dongjoon-hyun closed pull request #28118: [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/28118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27806: [SPARK-30914][CORE][DOC] Add version information to the configuration of UI
HyukjinKwon commented on issue #27806: [SPARK-30914][CORE][DOC] Add version information to the configuration of UI URL: https://github.com/apache/spark/pull/27806#issuecomment-609308954 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27745: [SPARK-30912][CORE][DOC] Add version information to the configuration of Streaming.scala
HyukjinKwon commented on issue #27745: [SPARK-30912][CORE][DOC] Add version information to the configuration of Streaming.scala URL: https://github.com/apache/spark/pull/27745#issuecomment-609306546 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History
HyukjinKwon commented on issue #27751: [SPARK-30891][CORE][DOC] Add version information to the configuration of History URL: https://github.com/apache/spark/pull/27751#issuecomment-609306424 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27734: [SPARK-30908][CORE][DOC] Add version information to the configuration of Kryo
HyukjinKwon commented on issue #27734: [SPARK-30908][CORE][DOC] Add version information to the configuration of Kryo URL: https://github.com/apache/spark/pull/27734#issuecomment-609306319 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27783: [SPARK-30889][SPARK-30913][CORE][DOC] Add version information to the configuration of Tests.scala and Worker
HyukjinKwon commented on issue #27783: [SPARK-30889][SPARK-30913][CORE][DOC] Add version information to the configuration of Tests.scala and Worker URL: https://github.com/apache/spark/pull/27783#issuecomment-609306746 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27674: [SPARK-30888][CORE][DOC] Add version information to the configuration of Network
HyukjinKwon commented on issue #27674: [SPARK-30888][CORE][DOC] Add version information to the configuration of Network URL: https://github.com/apache/spark/pull/27674#issuecomment-609306202 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27704: [SPARK-30909][CORE][DOC] Add version information to the configuration of Python
HyukjinKwon commented on issue #27704: [SPARK-30909][CORE][DOC] Add version information to the configuration of Python URL: https://github.com/apache/spark/pull/27704#issuecomment-609306084 Merged to branch-3.0 too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
AmplabJenkins removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609301126 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120821/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
SparkQA removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609106084 **[Test build #120821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120821/testReport)** for PR 28123 at commit [`94720e0`](https://github.com/apache/spark/commit/94720e00d222939085a7a5fd584608b8cfb2fc47). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
AmplabJenkins removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609301098 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
AmplabJenkins commented on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609301126 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120821/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
AmplabJenkins commented on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609301098 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28118: [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core
HyukjinKwon commented on issue #28118: [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/28118#issuecomment-609300052 Merged to branch-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on issue #28116: [SPARK-30841][SQL][DOC][3.0] Add version information to the configuration of SQL
HyukjinKwon removed a comment on issue #28116: [SPARK-30841][SQL][DOC][3.0] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/28116#issuecomment-609298615 Merged to branch-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
AmplabJenkins removed a comment on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609298144 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120822/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28116: [SPARK-30841][SQL][DOC][3.0] Add version information to the configuration of SQL
HyukjinKwon closed pull request #28116: [SPARK-30841][SQL][DOC][3.0] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/28116 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable
AmplabJenkins commented on issue #28123: [WIP][SPARK-31350][SQL] Coalesce bucketed tables for join if applicable URL: https://github.com/apache/spark/pull/28123#issuecomment-609298144 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120822/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
