[GitHub] viirya commented on a change in pull request #23272: [SPARK-26265][Core] Fix deadlock in BytesToBytesMap.MapIterator when locking both BytesToBytesMap.MapIterator and TaskMemoryManager
viirya commented on a change in pull request #23272: [SPARK-26265][Core] Fix
deadlock in BytesToBytesMap.MapIterator when locking both
BytesToBytesMap.MapIterator and TaskMemoryManager
URL: https://github.com/apache/spark/pull/23272#discussion_r240503360
##
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##
@@ -283,6 +290,9 @@ private void advanceToNextPage() {
}
}
}
+ if (pageToFree != null) {
+freePage(pageToFree);
Review comment:
The page to free (`currentPage`) is removed from `dataPages` and advanced to
next page when locking on the `MapIterator` object. The locking will prevent
the same page to be freed by calling `spill` at another consumer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] viirya commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
viirya commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE URL: https://github.com/apache/spark/pull/23213#issuecomment-446104710 I think wholeStageCodegen doesn't disallow using those objects in interpreted mode. The objects can be in interpreted mode if it rolls back from codegen in case of compilation error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446103258 @vanzin Thanks for the detailed review! Addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] LucaCanali commented on issue #22279: [SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics
LucaCanali commented on issue #22279: [SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics URL: https://github.com/apache/spark/pull/22279#issuecomment-446102201 Thanks @vanzin for looking at this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
AmplabJenkins removed a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446100267 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
SparkQA removed a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446096365 **[Test build #99955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99955/testReport)** for PR 23260 at commit [`dbeade7`](https://github.com/apache/spark/commit/dbeade7e41f861c9240c70058796293b239db96c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
AmplabJenkins removed a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446100271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99955/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
AmplabJenkins commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446100267 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
SparkQA commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446100212 **[Test build #99955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99955/testReport)** for PR 23260 at commit [`dbeade7`](https://github.com/apache/spark/commit/dbeade7e41f861c9240c70058796293b239db96c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
AmplabJenkins commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446100271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99955/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins removed a comment on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446097111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99944/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins removed a comment on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446097107 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins commented on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446097107 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins commented on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446097111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99944/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
felixcheung commented on a change in pull request #23072:
[SPARK-19827][R]spark.ml R API for PIC
URL: https://github.com/apache/spark/pull/23072#discussion_r240491948
##
File path: R/pkg/R/mllib_clustering.R
##
@@ -610,3 +616,59 @@ setMethod("write.ml", signature(object = "LDAModel", path
= "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
Review comment:
remove empty line - empty is significant in roxygen2
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
felixcheung commented on a change in pull request #23072:
[SPARK-19827][R]spark.ml R API for PIC
URL: https://github.com/apache/spark/pull/23072#discussion_r240492041
##
File path: R/pkg/R/mllib_clustering.R
##
@@ -610,3 +616,59 @@ setMethod("write.ml", signature(object = "LDAModel", path
= "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data a SparkDataFrame.
+#' @param k the number of clusters to create.
+#' @param initMode the initialization algorithm.
Review comment:
add `One of "random", "degree"`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
felixcheung commented on a change in pull request #23072:
[SPARK-19827][R]spark.ml R API for PIC
URL: https://github.com/apache/spark/pull/23072#discussion_r240492482
##
File path: R/pkg/R/mllib_clustering.R
##
@@ -610,3 +616,59 @@ setMethod("write.ml", signature(object = "LDAModel", path
= "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data a SparkDataFrame.
+#' @param k the number of clusters to create.
+#' @param initMode the initialization algorithm.
+#' @param maxIter the maximum number of iterations.
+#' @param sourceCol the name of the input column for source vertex IDs.
+#' @param destinationCol the name of the input column for destination vertex
IDs
+#' @param weightCol weight column name. If this is not set or \code{NULL},
+#' we treat all instance weights as 1.0.
+#' @param ... additional argument(s) passed to the method.
+#' @return A dataset that contains columns of vertex id and the corresponding
cluster for the id.
+#' The schema of it will be:
+#' \code{id: Long}
+#' \code{cluster: Int}
Review comment:
mm, this won't format correctly - roxygen strips all the whitespaces
also Long and Int is not a proper type in R
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
felixcheung commented on a change in pull request #23072:
[SPARK-19827][R]spark.ml R API for PIC
URL: https://github.com/apache/spark/pull/23072#discussion_r240492887
##
File path: R/pkg/R/mllib_clustering.R
##
@@ -610,3 +616,59 @@ setMethod("write.ml", signature(object = "LDAModel", path
= "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data a SparkDataFrame.
+#' @param k the number of clusters to create.
+#' @param initMode the initialization algorithm.
+#' @param maxIter the maximum number of iterations.
+#' @param sourceCol the name of the input column for source vertex IDs.
+#' @param destinationCol the name of the input column for destination vertex
IDs
+#' @param weightCol weight column name. If this is not set or \code{NULL},
+#' we treat all instance weights as 1.0.
+#' @param ... additional argument(s) passed to the method.
+#' @return A dataset that contains columns of vertex id and the corresponding
cluster for the id.
+#' The schema of it will be:
+#' \code{id: Long}
+#' \code{cluster: Int}
+#' @rdname spark.powerIterationClustering
+#' @aliases
assignClusters,PowerIterationClustering-method,SparkDataFrame-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0),
+#'list(1L, 2L, 1.0), list(3L, 4L, 1.0),
+#'list(4L, 0L, 0.1)),
+#' schema = c("src", "dst", "weight"))
+#' clusters <- spark.assignClusters(df, initMode="degree", weightCol="weight")
Review comment:
space around `=` as style
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule
SparkQA removed a comment on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446057878 **[Test build #99944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99944/testReport)** for PR 23273 at commit [`dfd0f71`](https://github.com/apache/spark/commit/dfd0f71afb8d95253ea4f64d00cea53c306b6e1c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
felixcheung commented on a change in pull request #23072:
[SPARK-19827][R]spark.ml R API for PIC
URL: https://github.com/apache/spark/pull/23072#discussion_r240493499
##
File path: R/pkg/R/mllib_clustering.R
##
@@ -610,3 +616,59 @@ setMethod("write.ml", signature(object = "LDAModel", path
= "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data a SparkDataFrame.
+#' @param k the number of clusters to create.
+#' @param initMode the initialization algorithm.
+#' @param maxIter the maximum number of iterations.
+#' @param sourceCol the name of the input column for source vertex IDs.
+#' @param destinationCol the name of the input column for destination vertex
IDs
+#' @param weightCol weight column name. If this is not set or \code{NULL},
+#' we treat all instance weights as 1.0.
+#' @param ... additional argument(s) passed to the method.
+#' @return A dataset that contains columns of vertex id and the corresponding
cluster for the id.
+#' The schema of it will be:
+#' \code{id: Long}
+#' \code{cluster: Int}
+#' @rdname spark.powerIterationClustering
+#' @aliases
assignClusters,PowerIterationClustering-method,SparkDataFrame-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0),
+#'list(1L, 2L, 1.0), list(3L, 4L, 1.0),
+#'list(4L, 0L, 0.1)),
+#' schema = c("src", "dst", "weight"))
+#' clusters <- spark.assignClusters(df, initMode="degree", weightCol="weight")
+#' showDF(clusters)
+#' }
+#' @note spark.assignClusters(SparkDataFrame) since 3.0.0
+setMethod("spark.assignClusters",
+ signature(data = "SparkDataFrame"),
+ function(data, k = 2L, initMode = c("random", "degree"), maxIter =
20L,
+sourceCol = "src", destinationCol = "dst", weightCol = NULL) {
+if (!is.numeric(k) || k < 1) {
+ stop("k should be a number with value >= 1.")
+}
+if (!is.integer(maxIter) || maxIter <= 0) {
Review comment:
if maxIter should in integer, should we check k is also integer? it;s fixed
when it is passed, so just a minor consistency on value check
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
felixcheung commented on a change in pull request #23072:
[SPARK-19827][R]spark.ml R API for PIC
URL: https://github.com/apache/spark/pull/23072#discussion_r240492789
##
File path: R/pkg/R/mllib_clustering.R
##
@@ -610,3 +616,59 @@ setMethod("write.ml", signature(object = "LDAModel", path
= "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data a SparkDataFrame.
+#' @param k the number of clusters to create.
+#' @param initMode the initialization algorithm.
+#' @param maxIter the maximum number of iterations.
+#' @param sourceCol the name of the input column for source vertex IDs.
+#' @param destinationCol the name of the input column for destination vertex
IDs
+#' @param weightCol weight column name. If this is not set or \code{NULL},
+#' we treat all instance weights as 1.0.
+#' @param ... additional argument(s) passed to the method.
+#' @return A dataset that contains columns of vertex id and the corresponding
cluster for the id.
+#' The schema of it will be:
+#' \code{id: Long}
+#' \code{cluster: Int}
+#' @rdname spark.powerIterationClustering
+#' @aliases
assignClusters,PowerIterationClustering-method,SparkDataFrame-method
Review comment:
wait, this aliases doesn't make sense. could you test if `?assignClusters`
in R shell if this works?
this should be `@aliases spark.assignClusters,SparkDataFrame-method`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule
SparkQA commented on issue #23273: [SPARK-25212][SQL][FOLLOWUP][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446096788 **[Test build #99944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99944/testReport)** for PR 23273 at commit [`dfd0f71`](https://github.com/apache/spark/commit/dfd0f71afb8d95253ea4f64d00cea53c306b6e1c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr
SparkQA commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446096365 **[Test build #99955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99955/testReport)** for PR 23260 at commit [`dbeade7`](https://github.com/apache/spark/commit/dbeade7e41f861c9240c70058796293b239db96c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446094075 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
SparkQA commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446094215 **[Test build #99954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99954/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446094078 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5958/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446094075 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] KyleLi1985 commented on a change in pull request #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
KyleLi1985 commented on a change in pull request #23271: [SPARK-26318][SQL]
Enhance function merge performance in Row
URL: https://github.com/apache/spark/pull/23271#discussion_r240491652
##
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
##
@@ -58,8 +58,21 @@ object Row {
* Merge multiple rows into a single row, one after another.
*/
def merge(rows: Row*): Row = {
-// TODO: Improve the performance of this if used in performance critical
part.
-new GenericRow(rows.flatMap(_.toSeq).toArray)
+val size = rows.size
+var number = 0
+for (i <- 0 until size) {
+ number = number + rows(i).size
+}
+val container = Array.ofDim[Any](number)
+var n = 0
+for (i <- 0 until size) {
+ val subSize = rows(i).size
+ for (j <- 0 until subSize) {
+container(n) = rows(i)(j)
+n = n + 1
+ }
+}
+new GenericRow(container)
Review comment:
definitely, It is important
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446094078 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5958/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] KyleLi1985 commented on a change in pull request #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
KyleLi1985 commented on a change in pull request #23271: [SPARK-26318][SQL]
Enhance function merge performance in Row
URL: https://github.com/apache/spark/pull/23271#discussion_r240491672
##
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
##
@@ -58,8 +58,21 @@ object Row {
* Merge multiple rows into a single row, one after another.
*/
def merge(rows: Row*): Row = {
-// TODO: Improve the performance of this if used in performance critical
part.
-new GenericRow(rows.flatMap(_.toSeq).toArray)
+val size = rows.size
+var number = 0
+for (i <- 0 until size) {
+ number = number + rows(i).size
+}
+val container = Array.ofDim[Any](number)
+var n = 0
+for (i <- 0 until size) {
Review comment:
Only primitively use size, subSize, and number information and control the
container will improve the performance more.
up to
call 1 time Row.merge(row1) need 18064 millisecond
call 1 time Row.merge(rows:_*) need 25651 millisecond
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
HyukjinKwon commented on a change in pull request #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#discussion_r240489089 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/BadRecordException.scala ## @@ -20,6 +20,16 @@ package org.apache.spark.sql.catalyst.util import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.unsafe.types.UTF8String +/** + * Exception thrown when the underlying parser returns a partial result of parsing. + * @param partialResult the partial result of parsing a bad record. + * @param cause the actual exception about why the parser cannot return full result. + */ +case class PartialResultException( Review comment: I mean, we don't have to standardise the name but let's use another name that doesn't conflict with Java's libraries. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
HyukjinKwon commented on a change in pull request #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#discussion_r240488920 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/BadRecordException.scala ## @@ -20,6 +20,16 @@ package org.apache.spark.sql.catalyst.util import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.unsafe.types.UTF8String +/** + * Exception thrown when the underlying parser returns a partial result of parsing. + * @param partialResult the partial result of parsing a bad record. + * @param cause the actual exception about why the parser cannot return full result. + */ +case class PartialResultException( Review comment: Wait .. but let's just rename it if possible .. the cost of renaming is 0 but there are some benefits by that .. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446090350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99943/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446090350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99943/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446090346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446090346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
SparkQA removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446057021 **[Test build #99943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99943/testReport)** for PR 23269 at commit [`8de1bcc`](https://github.com/apache/spark/commit/8de1bcca55a8b0b1448841871c47abee8101d917). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
SparkQA commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-44608 **[Test build #99943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99943/testReport)** for PR 23269 at commit [`8de1bcc`](https://github.com/apache/spark/commit/8de1bcca55a8b0b1448841871c47abee8101d917). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
cloud-fan commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE URL: https://github.com/apache/spark/pull/23213#issuecomment-446089670 when wholeStageCogen is on, there is no way to avoid codegen, so codegenFactoryMode doesn't make difference. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon edited a comment on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
HyukjinKwon edited a comment on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE URL: https://github.com/apache/spark/pull/23213#issuecomment-446088412 Ah, I had the same question as https://github.com/apache/spark/pull/23213#issuecomment-444824164. It would be good to update PR description :-). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
HyukjinKwon commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE URL: https://github.com/apache/spark/pull/23213#issuecomment-446088412 Ah, I had the same question as https://github.com/apache/spark/pull/23213#issuecomment-444824164. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23248: [SPARK-26293][SQL] Cast exception when having python udf in subquery
cloud-fan commented on issue #23248: [SPARK-26293][SQL] Cast exception when having python udf in subquery URL: https://github.com/apache/spark/pull/23248#issuecomment-446086659 thanks, merging to master/2.4! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446086294 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99950/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] asfgit closed pull request #23248: [SPARK-26293][SQL] Cast exception when having python udf in subquery
asfgit closed pull request #23248: [SPARK-26293][SQL] Cast exception when
having python udf in subquery
URL: https://github.com/apache/spark/pull/23248
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/python/pyspark/sql/tests/test_udf.py
b/python/pyspark/sql/tests/test_udf.py
index ed298f724d551..12cf8c7de1dad 100644
--- a/python/pyspark/sql/tests/test_udf.py
+++ b/python/pyspark/sql/tests/test_udf.py
@@ -23,7 +23,7 @@
from pyspark import SparkContext
from pyspark.sql import SparkSession, Column, Row
-from pyspark.sql.functions import UserDefinedFunction
+from pyspark.sql.functions import UserDefinedFunction, udf
from pyspark.sql.types import *
from pyspark.sql.utils import AnalysisException
from pyspark.testing.sqlutils import ReusedSQLTestCase, test_compiled,
test_not_compiled_message
@@ -102,7 +102,6 @@ def test_udf_registration_return_type_not_none(self):
def test_nondeterministic_udf(self):
# Test that nondeterministic UDFs are evaluated only once in chained
UDF evaluations
-from pyspark.sql.functions import udf
import random
udf_random_col = udf(lambda: int(100 * random.random()),
IntegerType()).asNondeterministic()
self.assertEqual(udf_random_col.deterministic, False)
@@ -113,7 +112,6 @@ def test_nondeterministic_udf(self):
def test_nondeterministic_udf2(self):
import random
-from pyspark.sql.functions import udf
random_udf = udf(lambda: random.randint(6, 6),
IntegerType()).asNondeterministic()
self.assertEqual(random_udf.deterministic, False)
random_udf1 = self.spark.catalog.registerFunction("randInt",
random_udf)
@@ -132,7 +130,6 @@ def test_nondeterministic_udf2(self):
def test_nondeterministic_udf3(self):
# regression test for SPARK-23233
-from pyspark.sql.functions import udf
f = udf(lambda x: x)
# Here we cache the JVM UDF instance.
self.spark.range(1).select(f("id"))
@@ -144,7 +141,7 @@ def test_nondeterministic_udf3(self):
self.assertFalse(deterministic)
def test_nondeterministic_udf_in_aggregate(self):
-from pyspark.sql.functions import udf, sum
+from pyspark.sql.functions import sum
import random
udf_random_col = udf(lambda: int(100 * random.random()),
'int').asNondeterministic()
df = self.spark.range(10)
@@ -181,7 +178,6 @@ def test_multiple_udfs(self):
self.assertEqual(tuple(row), (6, 5))
def test_udf_in_filter_on_top_of_outer_join(self):
-from pyspark.sql.functions import udf
left = self.spark.createDataFrame([Row(a=1)])
right = self.spark.createDataFrame([Row(a=1)])
df = left.join(right, on='a', how='left_outer')
@@ -190,7 +186,6 @@ def test_udf_in_filter_on_top_of_outer_join(self):
def test_udf_in_filter_on_top_of_join(self):
# regression test for SPARK-18589
-from pyspark.sql.functions import udf
left = self.spark.createDataFrame([Row(a=1)])
right = self.spark.createDataFrame([Row(b=1)])
f = udf(lambda a, b: a == b, BooleanType())
@@ -199,7 +194,6 @@ def test_udf_in_filter_on_top_of_join(self):
def test_udf_in_join_condition(self):
# regression test for SPARK-25314
-from pyspark.sql.functions import udf
left = self.spark.createDataFrame([Row(a=1)])
right = self.spark.createDataFrame([Row(b=1)])
f = udf(lambda a, b: a == b, BooleanType())
@@ -211,7 +205,7 @@ def test_udf_in_join_condition(self):
def test_udf_in_left_outer_join_condition(self):
# regression test for SPARK-26147
-from pyspark.sql.functions import udf, col
+from pyspark.sql.functions import col
left = self.spark.createDataFrame([Row(a=1)])
right = self.spark.createDataFrame([Row(b=1)])
f = udf(lambda a: str(a), StringType())
@@ -223,7 +217,6 @@ def test_udf_in_left_outer_join_condition(self):
def test_udf_in_left_semi_join_condition(self):
# regression test for SPARK-25314
-from pyspark.sql.functions import udf
left = self.spark.createDataFrame([Row(a=1, a1=1, a2=1), Row(a=2,
a1=2, a2=2)])
right = self.spark.createDataFrame([Row(b=1, b1=1, b2=1)])
f = udf(lambda a, b: a == b, BooleanType())
@@ -236,7 +229,6 @@ def test_udf_in_left_semi_join_condition(self):
def test_udf_and_common_filter_in_join_condition(self):
# regression test for SPARK-25314
# test the complex scenario with both udf and common filter
-from pyspark.sql.functions import udf
left = self.spark.createDataFrame([Row(a=1, a1=1, a2=1), Row(a=2,
a1=2, a2=2)])
right =
[GitHub] AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446086294 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99950/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446086290 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446086290 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
SparkQA removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446081233 **[Test build #99950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99950/testReport)** for PR 22273 at commit [`8574291`](https://github.com/apache/spark/commit/8574291a0b84574626ca213bc6f95dc0db73b0ef). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
SparkQA commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446086071 **[Test build #99950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99950/testReport)** for PR 22273 at commit [`8574291`](https://github.com/apache/spark/commit/8574291a0b84574626ca213bc6f95dc0db73b0ef). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class HaveArrowTests(unittest.TestCase):` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB
dongjoon-hyun commented on a change in pull request #23243:
[SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB
URL: https://github.com/apache/spark/pull/23243#discussion_r240483099
##
File path: core/src/test/scala/org/apache/spark/deploy/worker/WorkerSuite.scala
##
@@ -243,4 +243,13 @@ class WorkerSuite extends SparkFunSuite with Matchers
with BeforeAndAfter {
ExecutorStateChanged("app1", 0, ExecutorState.EXITED, None, None))
assert(cleanupCalled.get() == value)
}
+ test("test initRegisteredExecutorsDB ") {
+val sparkConf = new SparkConf()
+Utils.loadDefaultSparkProperties(sparkConf)
+val securityManager = new SecurityManager(sparkConf)
+sparkConf.set(config.SHUFFLE_SERVICE_DB_ENABLED.key, "true")
+sparkConf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+sparkConf.set("spark.local.dir", "/tmp")
+val externalShuffleService = new ExternalShuffleService(sparkConf,
securityManager)
Review comment:
Does this test case fail without your patch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
AmplabJenkins removed a comment on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#issuecomment-446084116 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
AmplabJenkins removed a comment on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#issuecomment-446084120 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5957/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
AmplabJenkins commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#issuecomment-446084120 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5957/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
AmplabJenkins commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#issuecomment-446084116 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
SparkQA commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#issuecomment-446084058 **[Test build #99953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99953/testReport)** for PR 23253 at commit [`9ca9248`](https://github.com/apache/spark/commit/9ca9248ed3f9314747c1415bd19760c53019bf36). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB
dongjoon-hyun commented on a change in pull request #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#discussion_r240482510 ## File path: core/src/test/scala/org/apache/spark/deploy/worker/WorkerSuite.scala ## @@ -19,20 +19,20 @@ package org.apache.spark.deploy.worker import java.util.concurrent.atomic.AtomicBoolean import java.util.function.Supplier - Review comment: Please execute `dev/scalastyle` to check the coding style. You should not remove this blank. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB
dongjoon-hyun commented on a change in pull request #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#discussion_r240482510 ## File path: core/src/test/scala/org/apache/spark/deploy/worker/WorkerSuite.scala ## @@ -19,20 +19,20 @@ package org.apache.spark.deploy.worker import java.util.concurrent.atomic.AtomicBoolean import java.util.function.Supplier - Review comment: Please execute `dev/scalastyle` to check the coding style. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446083193 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB
dongjoon-hyun commented on issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-446083308 Hi, @weixiuli . You can use `[CORE]` instead of `[ExternalShuffleService]`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun edited a comment on issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB
dongjoon-hyun edited a comment on issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-446083308 Hi, @weixiuli . You can use `[CORE]` instead of `[ExternalShuffleService]` in the PR title. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins removed a comment on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446083198 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5955/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446083193 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records
HyukjinKwon commented on issue #23253: [SPARK-26303][SQL] Return partial results for bad JSON records URL: https://github.com/apache/spark/pull/23253#issuecomment-446083244 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446083128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5956/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
SparkQA commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446083119 **[Test build #99951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99951/testReport)** for PR 23269 at commit [`a46d18e`](https://github.com/apache/spark/commit/a46d18e2a6ae822a1e1d903e54ab928096cb2339). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446083121 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446083128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5956/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
SparkQA commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446083138 **[Test build #99952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99952/testReport)** for PR 17085 at commit [`0cb2daf`](https://github.com/apache/spark/commit/0cb2daf35888d80c5c223e16505354571d87d383). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
AmplabJenkins commented on issue #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#issuecomment-446083198 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5955/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446083121 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
HyukjinKwon commented on issue #23271: [SPARK-26318][SQL] Enhance function merge performance in Row URL: https://github.com/apache/spark/pull/23271#issuecomment-446082473 +1 for deprecation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
AmplabJenkins removed a comment on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446082207 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
AmplabJenkins commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446082209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5954/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23280: [MINOR][SQL] Some errors in the notes.
dongjoon-hyun commented on a change in pull request #23280: [MINOR][SQL] Some
errors in the notes.
URL: https://github.com/apache/spark/pull/23280#discussion_r240481288
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
##
@@ -101,7 +101,7 @@ package object expressions {
StructType(attrs.map(a => StructField(a.name, a.dataType, a.nullable,
a.metadata)))
}
-// It's possible that `attrs` is a linked list, which can lead to bad
O(n^2) loops when
Review comment:
Ya. The original sentence has a different context at that time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] JkSelf commented on a change in pull request #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation
JkSelf commented on a change in pull request #23269: [SPARK-26316] Revert hash join metrics in spark 21052 that causes performance degradation URL: https://github.com/apache/spark/pull/23269#discussion_r240481284 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala ## @@ -62,8 +62,7 @@ case class HashAggregateExec( "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"), "peakMemory" -> SQLMetrics.createSizeMetric(sparkContext, "peak memory"), "spillSize" -> SQLMetrics.createSizeMetric(sparkContext, "spill size"), -"aggTime" -> SQLMetrics.createTimingMetric(sparkContext, "aggregate time"), -"avgHashProbe" -> SQLMetrics.createAverageMetric(sparkContext, "avg hash probe")) +"aggTime" -> SQLMetrics.createTimingMetric(sparkContext, "aggregate time")) Review comment: Yes, updated. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
AmplabJenkins commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446082207 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
AmplabJenkins removed a comment on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446082209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5954/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446081262 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins removed a comment on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446081265 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5953/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23277: [SPARK-26327][SQL] Metrics in FileSourceScanExec not update correctly
HyukjinKwon commented on issue #23277: [SPARK-26327][SQL] Metrics in FileSourceScanExec not update correctly URL: https://github.com/apache/spark/pull/23277#issuecomment-446081444 Looks fine to me This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446081262 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] CarolinePeng commented on a change in pull request #23280: [MINOR][SQL] Some errors in the notes.
CarolinePeng commented on a change in pull request #23280: [MINOR][SQL] Some
errors in the notes.
URL: https://github.com/apache/spark/pull/23280#discussion_r240480174
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
##
@@ -101,7 +101,7 @@ package object expressions {
StructType(attrs.map(a => StructField(a.name, a.dataType, a.nullable,
a.metadata)))
}
-// It's possible that `attrs` is a linked list, which can lead to bad
O(n^2) loops when
Review comment:
The cost time to access the linked list using ordinal is O (n), but I do not
combine BindReferences to analyze the cost time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
SparkQA commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446081221 **[Test build #99949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99949/testReport)** for PR 23251 at commit [`b1e71ee`](https://github.com/apache/spark/commit/b1e71ee7a723d63f1cf3c0754f2372eb185439d3). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
SparkQA commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446081233 **[Test build #99950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99950/testReport)** for PR 22273 at commit [`8574291`](https://github.com/apache/spark/commit/8574291a0b84574626ca213bc6f95dc0db73b0ef). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
AmplabJenkins commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446081265 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5953/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
dongjoon-hyun commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446081142 cc @tdas , too. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call
dongjoon-hyun commented on issue #23251: [SPARK-26300][SS] Remove a redundant `checkForStreaming` call URL: https://github.com/apache/spark/pull/23251#issuecomment-446081024 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23277: [SPARK-26327][SQL] Metrics in FileSourceScanExec not update correctly
HyukjinKwon commented on a change in pull request #23277: [SPARK-26327][SQL]
Metrics in FileSourceScanExec not update correctly
URL: https://github.com/apache/spark/pull/23277#discussion_r240480046
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##
@@ -316,7 +313,7 @@ case class FileSourceScanExec(
override lazy val metrics =
Map("numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of
output rows"),
"numFiles" -> SQLMetrics.createMetric(sparkContext, "number of files"),
- "metadataTime" -> SQLMetrics.createMetric(sparkContext, "metadata time
(ms)"),
+ "fileListingTime" -> SQLMetrics.createMetric(sparkContext, "file listing
time (ms)"),
Review comment:
Yea, please fix PR description and title accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] BryanCutler commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run
BryanCutler commented on issue #22273: [SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run URL: https://github.com/apache/spark/pull/22273#issuecomment-446080278 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
SparkQA removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446078542 **[Test build #99948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99948/testReport)** for PR 17085 at commit [`0480721`](https://github.com/apache/spark/commit/04807214d8694dcff7a2fe042457934e67eb8d57). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079818 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079818 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
SparkQA commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079811 **[Test build #99948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99948/testReport)** for PR 17085 at commit [`0480721`](https://github.com/apache/spark/commit/04807214d8694dcff7a2fe042457934e67eb8d57). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079415 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99947/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
SparkQA removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446077714 **[Test build #99947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99947/testReport)** for PR 17085 at commit [`0de3209`](https://github.com/apache/spark/commit/0de3209fa65b6391999668bf9e65042fefd27da5). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins removed a comment on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079413 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
SparkQA commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079407 **[Test build #99947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99947/testReport)** for PR 17085 at commit [`0de3209`](https://github.com/apache/spark/commit/0de3209fa65b6391999668bf9e65042fefd27da5). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator
AmplabJenkins commented on issue #17085: [SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator URL: https://github.com/apache/spark/pull/17085#issuecomment-446079415 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99947/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
