[GitHub] spark issue #18310: [SPARK-21103][SQL] QueryPlanConstraints should be part o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18310 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18313: [SPARK-21087] [ML] CrossValidator, TrainValidationSplit ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18313 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18313: [SPARK-21087] [ML] CrossValidator, TrainValidationSplit ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18313 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78085/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18310: [SPARK-21103][SQL] QueryPlanConstraints should be part o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18310 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78087/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18106 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78080/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18046: [SPARK-20749][SQL] Built-in SQL Function Support - all v...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18046 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78083/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18284 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78082/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18046: [SPARK-20749][SQL] Built-in SQL Function Support - all v...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18046 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18106 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18284 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18284 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78081/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18284 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18310: [SPARK-21103][SQL] QueryPlanConstraints should be part o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18310 **[Test build #3798 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3798/testReport)** for PR 18310 at commit [`7040877`](https://github.com/apache/spark/commit/704087798dd4e451fb3bb3caab0cdadd72ae19e5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18301 **[Test build #78088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78088/testReport)** for PR 18301 at commit [`bf4618a`](https://github.com/apache/spark/commit/bf4618a48a3d4abdedb6520e2b0405788c65a842). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18284 I can't find the test failed message, seems like unrelated errors. Can you trigger jenkins test again, because I'm not in the whitelist. Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18025#discussion_r122133379
--- Diff: R/pkg/R/functions.R ---
@@ -2254,18 +2198,12 @@ setMethod("approxCountDistinct",
column(jc)
})
-#' Count Distinct Values
+#' @section Details:
+#' \code{countDistinct}: Returns the number of distinct items in a group.
#'
-#' @param x Column to compute on
-#' @param ... other columns
--- End diff --
this is the example perhaps we are losing some details by grouping these
functions - instead of having a line that says "other columns" now we have
"other arguments" which is perhaps less clear?
thought?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18025#discussion_r122133897
--- Diff: R/pkg/R/generics.R ---
@@ -919,10 +920,9 @@ setGeneric("array_contains", function(x, value) {
standardGeneric("array_contain
#' @export
setGeneric("ascii", function(x) { standardGeneric("ascii") })
-#' @param x Column to compute on or a GroupedData object.
--- End diff --
I think we need to add this back to GroupedData, since that is now in a
different rd?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18025#discussion_r122133612
--- Diff: R/pkg/R/functions.R ---
@@ -3948,26 +3869,18 @@ setMethod("grouping_bit",
column(jc)
})
-#' grouping_id
-#'
-#' Returns the level of grouping.
-#'
+#' @section Details:
+#' \code{grouping_id}: Returns the level of grouping.
#' Equals to \code{
#' grouping_bit(c1) * 2^(n - 1) + grouping_bit(c2) * 2^(n - 2) + ... +
grouping_bit(cn)
#' }
#'
-#' @param x Column to compute on
-#' @param ... additional Column(s) (optional).
--- End diff --
ditto here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18025#discussion_r122133805
--- Diff: R/pkg/R/generics.R ---
@@ -1403,20 +1416,25 @@ setGeneric("unix_timestamp", function(x, format) {
standardGeneric("unix_timesta
#' @export
setGeneric("upper", function(x) { standardGeneric("upper") })
-#' @rdname var
+#' @rdname column_aggregate_functions
+#' @param y,na.rm,use currently not used.
--- End diff --
hmm, this is in `var` but also in `sd` too - perhaps should move to
column_aggregate_functions
in fact, in your screen shot they are both listed
https://github.com/apache/spark/pull/18025#issuecomment-303877945
`y,na.rm,use` and then `na.rm`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18025#discussion_r122131981
--- Diff: R/pkg/R/functions.R ---
@@ -85,17 +100,20 @@ setMethod("acos",
column(jc)
})
-#' Returns the approximate number of distinct items in a group
+#' @section Details:
--- End diff --
would it be more concise to do `@details`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18025: [SPARK-20889][SparkR] Grouped documentation for A...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18025#discussion_r122132206
--- Diff: R/pkg/R/functions.R ---
@@ -85,17 +100,20 @@ setMethod("acos",
column(jc)
})
-#' Returns the approximate number of distinct items in a group
+#' @section Details:
+#' \code{approxCountDistinct}: Returns the approximate number of distinct
items in a group.
#'
-#' Returns the approximate number of distinct items in a group. This is a
column
-#' aggregate function.
-#'
-#' @rdname approxCountDistinct
-#' @name approxCountDistinct
-#' @return the approximate number of distinct items in a group.
+#' @rdname column_aggregate_functions
#' @export
-#' @aliases approxCountDistinct,Column-method
-#' @examples \dontrun{approxCountDistinct(df$c)}
+#' @aliases approxCountDistinct approxCountDistinct,Column-method
+#' @examples
+#'
--- End diff --
extra newline?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18301 Maybe just min and max? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18304: [SPARK-21098] Add lineseparator parameter to csv ...
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122135698
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val quote = getChar("quote", '\"')
val escape = getChar("escape", '\\')
val comment = getChar("comment", '\u')
+ val lineSeparator = parameters.getOrElse("lineSeparator", "\n")
--- End diff --
Hi, thanks for pointing this out, I wasn't aware of this. Looking at it
though, how would you propose fixing this? We would need a `writerSettings`
instance to fetch the univocity default I think. Also, looking at the other
parameters, I also see hardcoded values there (for instance `val quote =
getChar("quote", '\"')`). Personally, I like the idea of being able to see the
default used directly in Spark, to prevent changes in the univocity lib from
impacting Spark usage (and requiring confusion when using).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18301 **[Test build #78089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78089/testReport)** for PR 18301 at commit [`438d0e1`](https://github.com/apache/spark/commit/438d0e17fe3efbdff63ee51a2e4591cf61efd5a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18128: [SPARK-20906][SparkR]:Constrained Logistic Regres...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18128#discussion_r122135013
--- Diff: R/pkg/R/mllib_classification.R ---
@@ -202,6 +202,20 @@ function(object, path, overwrite = FALSE) {
#' @param aggregationDepth The depth for treeAggregate (greater than or
equal to 2). If the dimensions of features
#' or the number of partitions are large, this
param could be adjusted to a larger size.
#' This is an expert parameter. Default value
should be good for most cases.
+#' @param lowerBoundsOnCoefficients The lower bounds on coefficients if
fitting under bound constrained optimization.
+#' The bound matrix must be compatible
with the shape (1, number of features) for binomial
+#' regression, or (number of classes,
number of features) for multinomial regression.
+#' It is a R matrix.
+#' @param upperBoundsOnCoefficients The upper bounds on coefficients if
fitting under bound constrained optimization.
+#' The bound matrix must be compatible
with the shape (1, number of features) for binomial
+#' regression, or (number of classes,
number of features) for multinomial regression.
+#' It is a R matrix.
+#' @param lowerBoundsOnIntercepts The lower bounds on intercepts if
fitting under bound constrained optimization.
+#'The bounds vector size must be equal
with 1 for binomial regression, or the number
+#'of classes for multinomial regression.
+#' @param upperBoundsOnIntercepts The upper bounds on intercepts if
fitting under bound constrained optimization.
+#'The bound vector size must be equal with
1 for binomial regression, or the number
--- End diff --
ditto here `size must be equal with 1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18128: [SPARK-20906][SparkR]:Constrained Logistic Regres...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18128#discussion_r122134992
--- Diff: R/pkg/R/mllib_classification.R ---
@@ -202,6 +202,20 @@ function(object, path, overwrite = FALSE) {
#' @param aggregationDepth The depth for treeAggregate (greater than or
equal to 2). If the dimensions of features
#' or the number of partitions are large, this
param could be adjusted to a larger size.
#' This is an expert parameter. Default value
should be good for most cases.
+#' @param lowerBoundsOnCoefficients The lower bounds on coefficients if
fitting under bound constrained optimization.
+#' The bound matrix must be compatible
with the shape (1, number of features) for binomial
+#' regression, or (number of classes,
number of features) for multinomial regression.
+#' It is a R matrix.
+#' @param upperBoundsOnCoefficients The upper bounds on coefficients if
fitting under bound constrained optimization.
+#' The bound matrix must be compatible
with the shape (1, number of features) for binomial
+#' regression, or (number of classes,
number of features) for multinomial regression.
+#' It is a R matrix.
+#' @param lowerBoundsOnIntercepts The lower bounds on intercepts if
fitting under bound constrained optimization.
+#'The bounds vector size must be equal
with 1 for binomial regression, or the number
--- End diff --
`size must be equal with 1` - should this be `size must be equal to 1`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18128: [SPARK-20906][SparkR]:Constrained Logistic Regres...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18128#discussion_r122135628
--- Diff: R/pkg/R/mllib_classification.R ---
@@ -239,21 +253,64 @@ function(object, path, overwrite = FALSE) {
setMethod("spark.logit", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, regParam = 0.0, elasticNetParam = 0.0,
maxIter = 100,
tol = 1E-6, family = "auto", standardization = TRUE,
- thresholds = 0.5, weightCol = NULL, aggregationDepth =
2) {
+ thresholds = 0.5, weightCol = NULL, aggregationDepth =
2,
+ lowerBoundsOnCoefficients = NULL,
upperBoundsOnCoefficients = NULL,
+ lowerBoundsOnIntercepts = NULL, upperBoundsOnIntercepts
= NULL) {
formula <- paste(deparse(formula), collapse = "")
+row <- 0
+col <- 0
if (!is.null(weightCol) && weightCol == "") {
weightCol <- NULL
} else if (!is.null(weightCol)) {
weightCol <- as.character(weightCol)
}
+if (!is.null(lowerBoundsOnIntercepts)) {
+lowerBoundsOnIntercepts <-
as.array(lowerBoundsOnIntercepts)
+}
+
+if (!is.null(upperBoundsOnIntercepts)) {
+upperBoundsOnIntercepts <-
as.array(upperBoundsOnIntercepts)
+}
+
+if (!is.null(lowerBoundsOnCoefficients)) {
+ if (class(lowerBoundsOnCoefficients) != "matrix") {
+stop("lowerBoundsOnCoefficients must be a matrix.")
+ }
+ row <- nrow(lowerBoundsOnCoefficients)
+ col <- ncol(lowerBoundsOnCoefficients)
+ lowerBoundsOnCoefficients <-
as.array(as.vector(lowerBoundsOnCoefficients))
+}
+
+if (!is.null(upperBoundsOnCoefficients)) {
+ if (class(upperBoundsOnCoefficients) != "matrix") {
+stop("upperBoundsOnCoefficients must be a matrix.")
+ }
+
+ if (!is.null(lowerBoundsOnCoefficients) & (row !=
nrow(upperBoundsOnCoefficients)
+| col != ncol(upperBoundsOnCoefficients))) {
--- End diff --
while logically and semantically correct here, might prefer `&&` and `||`
instead of `&` and `|` here for programmability/correctness in case something
becomes length > 1 here later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18291: [SPARK-20892][SparkR] Add SQL trunc function to S...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18291#discussion_r122136538
--- Diff: R/pkg/NAMESPACE ---
@@ -357,6 +357,7 @@ exportMethods("%<=>%",
"to_utc_timestamp",
"translate",
"trim",
+ "trunc",
--- End diff --
and looks like it has a generic already?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18291: [SPARK-20892][SparkR] Add SQL trunc function to S...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18291#discussion_r122136452
--- Diff: R/pkg/R/functions.R ---
@@ -4015,3 +4015,31 @@ setMethod("input_file_name", signature("missing"),
jc <- callJStatic("org.apache.spark.sql.functions",
"input_file_name")
column(jc)
})
+
+#' trunc
+#'
+#' Returns date truncated to the unit specified by the format.
+#'
+#' @param x Column to compute on.
+#' @param format string used for specify the truncation method. For
example, "year", "",
+#' "yy" for truncate by year, or "month", "mon", "mm" for truncate by
month.
+#'
+#' @rdname trunc
+#' @name trunc
+#' @family date time functions
+#' @aliases trunc,Column-method
+#' @export
+#' @examples
+#' \dontrun{
+#' trunc(df$c, "year")
+#' trunc(df$c, "yy")
+#' trunc(df$c, "month")
+#' trunc(df$c, "mon")
+#' }
+#' @note trunc since 2.3.0
+setMethod("trunc",
+ signature(x = "Column"),
+ function(x, format) {
+jc <- callJStatic("org.apache.spark.sql.functions", "trunc",
x@jc, format)
--- End diff --
I'd `as.character(format)` here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18291: [SPARK-20892][SparkR] Add SQL trunc function to S...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18291#discussion_r122136198
--- Diff: R/pkg/NAMESPACE ---
@@ -357,6 +357,7 @@ exportMethods("%<=>%",
"to_utc_timestamp",
"translate",
"trim",
+ "trunc",
--- End diff --
it doesn't mask base::trunc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should ...
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/18284#discussion_r122123445
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala
---
@@ -146,6 +154,17 @@ class TreeNodeSuite extends SparkFunSuite {
assert(actual === Dummy(None))
}
+ test("mapChildren should only works on children") {
+val children = Seq((Literal(1), Literal(2)))
+val nonChildren = Seq((Literal(3), Literal(4)))
+val before = SeqTupleExpression(children, nonChildren)
+val toZero: PartialFunction[Expression, Expression] = { case
Literal(_, _) => Literal(0) }
+val expect = SeqTupleExpression(Seq((Literal(0), Literal(0))),
nonChildren)
+
+val actual = before mapChildren toZero
+assert(actual === expect)
--- End diff --
Maybe better to use `.equals`? Although it will call Object's `.equals`
which is `==`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18283: [TEST][SPARKR][CORE] Fix broken SparkSubmitSuite
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/18283 I guess that script can be changed to use JAVA_HOME instead of blindly assume it's accessible... are these new machines in Jenkins? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should ...
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/18284#discussion_r122139485
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala
---
@@ -146,6 +154,17 @@ class TreeNodeSuite extends SparkFunSuite {
assert(actual === Dummy(None))
}
+ test("mapChildren should only works on children") {
+val children = Seq((Literal(1), Literal(2)))
+val nonChildren = Seq((Literal(3), Literal(4)))
+val before = SeqTupleExpression(children, nonChildren)
+val toZero: PartialFunction[Expression, Expression] = { case
Literal(_, _) => Literal(0) }
+val expect = SeqTupleExpression(Seq((Literal(0), Literal(0))),
nonChildren)
+
+val actual = before mapChildren toZero
+assert(actual === expect)
--- End diff --
I'm not sure if I understand it wrongly. `===` compared to `==` could
provide more information about the error. You can see the follow examples:
```
scala> assert(1 == 2)
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:156)
... 32 elided
```
```
scala> assert(1 === 2)
:12: error: value === is not a member of Int
assert(1 === 2)
```
And also you can check it in
https://stackoverflow.com/questions/10489548/what-is-the-triple-equals-operator-in-scala-koans
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18304: [SPARK-21098] Add lineseparator parameter to csv ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122141063
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val quote = getChar("quote", '\"')
val escape = getChar("escape", '\\')
val comment = getChar("comment", '\u')
+ val lineSeparator = parameters.getOrElse("lineSeparator", "\n")
--- End diff --
Yea, that's why I haven't proposed this change so far (for both hard corded
`\n` and configurable line separator).
For the hard corded `\n`, IMHO, reviving hard corded `\n` is probably fine
for now as consistent with JSON/TEXT datasources because It uses, after the
change I described above, OS-dependent newlline by default in Univocity -
https://github.com/uniVocity/univocity-parsers/blob/e34a149077147de7cafcafbbaf8a4310e6297584/src/main/java/com/univocity/parsers/common/Format.java#L78,
which is machine-dependent. If we are worried of changing this, we could use
`System.getProperty("line.separator")` or just not set via `Option`. This
probably does not block this PR.
For the configurable line separator, we are not able to read it back when
`wholeFile` (or `miltiLine` now) option is disabled (it's default and this
should be primary) because we are dependent of `LineRecordReader` which only
deals with `\n` or `\r\n`. We really should be able to read it back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18304: [SPARK-21098] Add lineseparator parameter to csv ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122142421
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val quote = getChar("quote", '\"')
val escape = getChar("escape", '\\')
val comment = getChar("comment", '\u')
+ val lineSeparator = parameters.getOrElse("lineSeparator", "\n")
--- End diff --
To cut this short, up to my knowledge, for the current status (please
correct me if I am wrong),
CSV read -> cover `\n` and `\r\n`
CSV read with `wholeFile` (`multiLine`) -> OS dependent newline (by
Univocity)
CSV write -> OS dependent newline (by Univocity)
JSON read -> cover `\n` and `\r\n`
JSON read with `wholeFile` (`multiLine`) -> N/A (it reads the whole file as
a single record for the current status)
JSON write -> `\n`
TEXT read -> cover `\n` and `\r\n`
TEXT write -> `\n`
For hardcorded newline, fixing it to `\n` is probably better. I wouldn't
mind if this PR is turned to fix this for 'CSV read with wholeFile (multiLine)'
and 'CSV write'.
For configurable line separator, it would be not configurable in this way
for 'CSV read'. We should be able to read it back what we write out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18228 **[Test build #78090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78090/testReport)** for PR 18228 at commit [`2136c1b`](https://github.com/apache/spark/commit/2136c1bfac33a32cd52780080aa48f7e25e7c463). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18304: [SPARK-21098] Add lineseparator parameter to csv ...
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122144727
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val quote = getChar("quote", '\"')
val escape = getChar("escape", '\\')
val comment = getChar("comment", '\u')
+ val lineSeparator = parameters.getOrElse("lineSeparator", "\n")
--- End diff --
Ah, I think I understand now, thanks.
Maybe I'm overlooking it, but I don't see the `wholeFile` option being used
in `CsvOptions` at all at the moment (nor `multiline`) ?
I'll take a further look at fixing this up considering the dependency on
`LineRecordReader` (which only supports the two line separators you mentioned)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18301 **[Test build #78091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78091/testReport)** for PR 18301 at commit [`69e8216`](https://github.com/apache/spark/commit/69e82169817bfaa7a80e046cf4e12ed883621920). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18284 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18266: [SPARK-20427][SQL] Read JDBC table use custom schema
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18266 **[Test build #78093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78093/testReport)** for PR 18266 at commit [`06881e8`](https://github.com/apache/spark/commit/06881e800c7ddd1e72069ae1c055d1dbd1f38bdc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18284 **[Test build #78092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78092/testReport)** for PR 18284 at commit [`9a9d8af`](https://github.com/apache/spark/commit/9a9d8afee0e4ef59bfd4faca66e106851d51822e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18301 The screenshot of UI showing float numbers instead of integer numbers. https://user-images.githubusercontent.com/68855/27172399-bc6c8390-51e7-11e7-854a-11963aeda216.png";> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18284 Nit: The PR title doesn't need to add ``. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/18106 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should ...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18284#discussion_r122151719
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala
---
@@ -61,6 +61,14 @@ case class ExpressionInMap(map: Map[String, Expression])
extends Expression with
override lazy val resolved = true
}
+case class SeqTupleExpression(sons: Seq[(Expression, Expression)],
+nonSons: Seq[(Expression, Expression)]) extends Expression with
Unevaluable {
--- End diff --
`Unevaluable` already extends `Expression`. I think you don't need to
extend `Expression` again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18284 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18106 **[Test build #78094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78094/testReport)** for PR 18106 at commit [`b391b6a`](https://github.com/apache/spark/commit/b391b6a3e51229e501982fe184685d7c2e185172). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18314: [SPARK-16251][SPARK-20200][Core][Test]Flaky test:...
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/18314 [SPARK-16251][SPARK-20200][Core][Test]Flaky test: org.apache.spark.rdd.LocalCheckpointSuite.missing checkpoint block fails with informative message ## What changes were proposed in this pull request? Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case. The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply(). ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiangxb1987/spark LocalCheckpointSuite Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18314.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18314 commit 7515a5bf6f3899081b528dc91951e286ac6a3e1c Author: Xingbo Jiang Date: 2017-06-15T08:52:22Z fix flaky test case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18314: [SPARK-16251][SPARK-20200][Core][Test]Flaky test: org.ap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18314 **[Test build #78095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78095/testReport)** for PR 18314 at commit [`7515a5b`](https://github.com/apache/spark/commit/7515a5bf6f3899081b528dc91951e286ac6a3e1c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18304: [SPARK-21098] Add lineseparator parameter to csv ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122156785
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val quote = getChar("quote", '\"')
val escape = getChar("escape", '\\')
val comment = getChar("comment", '\u')
+ val lineSeparator = parameters.getOrElse("lineSeparator", "\n")
--- End diff --
Ah, it was `wholeFile` but just renamed
https://github.com/apache/spark/commit/2051428173d8cd548702eb1a2e1c1ca76b8f2fd5
to `multiLine` four hours ago.
Yes, it would be wonderful if we can support it in reading too. If it is
difficult, it is fine (to me) to turn this PR for hard corded one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18314: [SPARK-16251][SPARK-20200][Core][Test]Flaky test:...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18314#discussion_r122157170
--- Diff:
core/src/test/scala/org/apache/spark/rdd/LocalCheckpointSuite.scala ---
@@ -168,6 +172,10 @@ class LocalCheckpointSuite extends SparkFunSuite with
LocalSparkContext {
// Collecting the RDD should now fail with an informative exception
val blockId = RDDBlockId(rdd.id, numPartitions - 1)
bmm.removeBlock(blockId)
+// Wait until the block has been removed successfully.
+eventually(timeout(1 seconds), interval(100 milliseconds)) {
+ assert(bmm.getBlockStatus(blockId).size == 0)
--- End diff --
Use isEmpty or ===
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18301 **[Test build #78088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78088/testReport)** for PR 18301 at commit [`bf4618a`](https://github.com/apache/spark/commit/bf4618a48a3d4abdedb6520e2b0405788c65a842). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78088/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18301 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18310: [SPARK-21103][SQL] QueryPlanConstraints should be part o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18310 **[Test build #3798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3798/testReport)** for PR 18310 at commit [`7040877`](https://github.com/apache/spark/commit/704087798dd4e451fb3bb3caab0cdadd72ae19e5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18301 **[Test build #78089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78089/testReport)** for PR 18301 at commit [`438d0e1`](https://github.com/apache/spark/commit/438d0e17fe3efbdff63ee51a2e4591cf61efd5a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17865 Thanks for working on this :) More complete documentation is certainly useful. Looking at the "How was this patch tested?" section you might also want to build the docs (there are instructions in `docs/README.md`) to see how stuff is formatted. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18301 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78089/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17435: [SPARK-20098][PYSPARK] dataType's typeName fix
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17435 Sorry for my delay finishing the review. This looks good pending Jenkins and if it passes I'll try and merge it when I get home. (so Jenkins looks good to test. Jenkins test this please. ) :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18271: [MINOR][DOCS] Improve Running R Tests docs
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18271 **[Test build #78096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78096/testReport)** for PR 18271 at commit [`ff6415f`](https://github.com/apache/spark/commit/ff6415fecfd1542eaf4eb35e1b2ba418da856761). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17435: [SPARK-20098][PYSPARK] dataType's typeName fix
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17435 **[Test build #78097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78097/testReport)** for PR 17435 at commit [`6bf11f0`](https://github.com/apache/spark/commit/6bf11f027a5117b82936f88918e13e89993f2108). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17435: [SPARK-20098][PYSPARK] dataType's typeName fix
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17435 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17435: [SPARK-20098][PYSPARK] dataType's typeName fix
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78097/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17435: [SPARK-20098][PYSPARK] dataType's typeName fix
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17435 **[Test build #78097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78097/testReport)** for PR 17435 at commit [`6bf11f0`](https://github.com/apache/spark/commit/6bf11f027a5117b82936f88918e13e89993f2108). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18303 retest please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to r...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16225#discussion_r122164228
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -1391,9 +1401,10 @@ def all_of_(xs):
"to_replace should be a float, int, long, string, list,
tuple, or dict. "
"Got {0}".format(type(to_replace)))
-if not isinstance(value, valid_types) and not
isinstance(to_replace, dict):
+if not isinstance(value, valid_types) and value is not None \
--- End diff --
So slightly jet lagged style question: would this be clearer if we just add
type(None) to L1398? I know PEP8 says we should only use `is` `is not` for
checking if something is none rather than depending on the implicit conversion
to boolean -- but since really checking the type here we aren't really in
danger of that. (This is just a suggestion to make it easier to read - if
others think its easier to read this way thats fine :)). @davies ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to r...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/16225#discussion_r122166101 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1423,7 +1434,9 @@ def all_of_(xs): subset = [subset] # Verify we were not passed in mixed type generics." -if not any(all_of_type(rep_dict.keys()) and all_of_type(rep_dict.values()) +if not any(all_of_type(rep_dict.keys()) + and (all_of_type(rep_dict.values()) +or list(rep_dict.values()).count(None) == len(rep_dict)) --- End diff -- In the Scala code null is allowed in to be the replacement value for some of the elements but not in the Python code. Is this intentional? If so we should document it clearly and expand on the error message bellow (otherwise we should make it more flexible). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to r...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/16225#discussion_r122164388 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1409,7 +1420,7 @@ def all_of_(xs): if isinstance(to_replace, (float, int, long, basestring)): to_replace = [to_replace] -if isinstance(value, (float, int, long, basestring)): +if isinstance(value, (float, int, long, basestring)) or value is None: --- End diff -- same as above --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18271: [MINOR][DOCS] Improve Running R Tests docs
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18271 **[Test build #78096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78096/testReport)** for PR 18271 at commit [`ff6415f`](https://github.com/apache/spark/commit/ff6415fecfd1542eaf4eb35e1b2ba418da856761). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18314: [SPARK-16251][SPARK-20200][Core][Test]Flaky test: org.ap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18314 **[Test build #78098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78098/testReport)** for PR 18314 at commit [`4c792b9`](https://github.com/apache/spark/commit/4c792b9cd09e8e41792a5c066d8a40fca6e17183). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18271: [MINOR][DOCS] Improve Running R Tests docs
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78096/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18271: [MINOR][DOCS] Improve Running R Tests docs
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18271 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18228 **[Test build #78090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78090/testReport)** for PR 18228 at commit [`2136c1b`](https://github.com/apache/spark/commit/2136c1bfac33a32cd52780080aa48f7e25e7c463). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Right(str: Expression, len: Expression)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18228 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78090/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18228 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18301 **[Test build #78091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78091/testReport)** for PR 18301 at commit [`69e8216`](https://github.com/apache/spark/commit/69e82169817bfaa7a80e046cf4e12ed883621920). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78091/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18301: [SPARK-21052][SQL] Add hash map metrics to join
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18301 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18302 **[Test build #78099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78099/testReport)** for PR 18302 at commit [`b423e6f`](https://github.com/apache/spark/commit/b423e6f2b5f3d56a5da289f186dcc5bcea09b08b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18302 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18302 **[Test build #78099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78099/testReport)** for PR 18302 at commit [`b423e6f`](https://github.com/apache/spark/commit/b423e6f2b5f3d56a5da289f186dcc5bcea09b08b). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18302 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78099/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18284 **[Test build #78092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78092/testReport)** for PR 18284 at commit [`9a9d8af`](https://github.com/apache/spark/commit/9a9d8afee0e4ef59bfd4faca66e106851d51822e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18284 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18284 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78092/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18302 **[Test build #78100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78100/testReport)** for PR 18302 at commit [`a8728f6`](https://github.com/apache/spark/commit/a8728f6e63c6bf37d45be18364f265cb69496376). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18302 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78100/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18302 **[Test build #78100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78100/testReport)** for PR 18302 at commit [`a8728f6`](https://github.com/apache/spark/commit/a8728f6e63c6bf37d45be18364f265cb69496376). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18302 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18302: [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18302 **[Test build #78101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78101/testReport)** for PR 18302 at commit [`d5e0eab`](https://github.com/apache/spark/commit/d5e0eab2bd13e3e597df7fa19cebe35a3f008139). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17451: [SPARK-19866][ML][PySpark] Add local version of Word2Vec...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17451 So I belive `._1()`, so very close just need do the function call on it @keypointt . Sorry for my slowness responding I've been traveling a lot lately. Let me know if you have any other questions :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17237: [SPARK-19852][PYSPARK][ML] Update Python API setHandleIn...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17237 gentle re-ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18282: [SPARK-21070][PYSPARK] Upgrade cloudpickle
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/18282 Jenkins, ok to test. Jenkins add to whitelist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18282: [SPARK-21070][PYSPARK] Upgrade cloudpickle
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18282 **[Test build #78102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78102/testReport)** for PR 18282 at commit [`b84222b`](https://github.com/apache/spark/commit/b84222b6ea660ce5ec1aedfee50e297b436eb824). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18106 **[Test build #78094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78094/testReport)** for PR 18106 at commit [`b391b6a`](https://github.com/apache/spark/commit/b391b6a3e51229e501982fe184685d7c2e185172). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18106 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18266: [SPARK-20427][SQL] Read JDBC table use custom schema
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18266 **[Test build #78093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78093/testReport)** for PR 18266 at commit [`06881e8`](https://github.com/apache/spark/commit/06881e800c7ddd1e72069ae1c055d1dbd1f38bdc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18106 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78094/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18266: [SPARK-20427][SQL] Read JDBC table use custom schema
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18266 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
