[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9769 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160452624 LGTM. Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160418496 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46847/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160418494 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160418454 **[Test build #46847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46847/consoleFull)** for PR 9769 at commit [`0b1af63`](https://github.com/apache/spark/commit/0b1af635a88a3ebe10c5f5ba0b007dfe936ab51b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160417514 **[Test build #46847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46847/consoleFull)** for PR 9769 at commit [`0b1af63`](https://github.com/apache/spark/commit/0b1af635a88a3ebe10c5f5ba0b007dfe936ab51b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9769#discussion_r46088623
--- Diff: R/pkg/R/DataFrame.R ---
@@ -700,25 +700,28 @@ setMethod("collect",
# data of complex type can be held. But getting a cell
from a column
# of list type returns a list instead of a vector. So for
columns of
# non-complex type, append them as vector.
+#
+# For columns of complex type, be careful to access them.
+# Get a column of complex type returns a list.
+# Get a cell from a column of complex type returns a list
instead of a vector.
col <- listCols[[colIndex]]
+colName <- dtypes[[colIndex]][[1]]
if (length(col) <= 0) {
- df[[names[colIndex]]] <- col
+ df[[colName]] <- col
} else {
- # TODO: more robust check on column of primitive types
- vec <- do.call(c, col)
- if (class(vec) != "list") {
-df[[names[colIndex]]] <- vec
+ colType <- dtypes[[colIndex]][[2]]
+ if (!is.null(PRIMITIVE_TYPES[[colType]]) && colType !=
"binary") {
--- End diff --
added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160373978 LGTM but for a minor comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9769#discussion_r46087941
--- Diff: R/pkg/R/DataFrame.R ---
@@ -700,25 +700,28 @@ setMethod("collect",
# data of complex type can be held. But getting a cell
from a column
# of list type returns a list instead of a vector. So for
columns of
# non-complex type, append them as vector.
+#
+# For columns of complex type, be careful to access them.
+# Get a column of complex type returns a list.
+# Get a cell from a column of complex type returns a list
instead of a vector.
col <- listCols[[colIndex]]
+colName <- dtypes[[colIndex]][[1]]
if (length(col) <= 0) {
- df[[names[colIndex]]] <- col
+ df[[colName]] <- col
} else {
- # TODO: more robust check on column of primitive types
- vec <- do.call(c, col)
- if (class(vec) != "list") {
-df[[names[colIndex]]] <- vec
+ colType <- dtypes[[colIndex]][[2]]
+ if (!is.null(PRIMITIVE_TYPES[[colType]]) && colType !=
"binary") {
--- End diff --
Could you add a comment here as well ? Something like `NOTE: "binary"
columns behave like complex types`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160023492 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160023496 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46792/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160023298 **[Test build #46792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46792/consoleFull)** for PR 9769 at commit [`f073c3a`](https://github.com/apache/spark/commit/f073c3aede9fd258d2354db04d5eae7c14e40c25). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160018492 **[Test build #46792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46792/consoleFull)** for PR 9769 at commit [`f073c3a`](https://github.com/apache/spark/commit/f073c3aede9fd258d2354db04d5eae7c14e40c25). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9769#discussion_r46012852
--- Diff: R/pkg/R/DataFrame.R ---
@@ -700,25 +700,28 @@ setMethod("collect",
# data of complex type can be held. But getting a cell
from a column
# of list type returns a list instead of a vector. So for
columns of
# non-complex type, append them as vector.
+#
+# For columns of complex type, be careful to access them.
+# Get a column of complex type returns a list.
+# Get a cell from a column of complex type returns a list
instead of a vector.
col <- listCols[[colIndex]]
+colName <- dtypes[[colIndex]][[1]]
if (length(col) <= 0) {
- df[[names[colIndex]]] <- col
+ df[[colName]] <- col
} else {
- # TODO: more robust check on column of primitive types
- vec <- do.call(c, col)
- if (class(vec) != "list") {
-df[[names[colIndex]]] <- vec
+ colType <- dtypes[[colIndex]][[2]]
+ if (!is.null(PRIMITIVE_TYPES[[colType]]) && colType !=
"binary") {
+vec <- do.call(c, col)
+stopifnot (class(vec) != "list")
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-160013831 looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9769#discussion_r46011950
--- Diff: R/pkg/R/DataFrame.R ---
@@ -700,25 +700,28 @@ setMethod("collect",
# data of complex type can be held. But getting a cell
from a column
# of list type returns a list instead of a vector. So for
columns of
# non-complex type, append them as vector.
+#
+# For columns of complex type, be careful to access them.
+# Get a column of complex type returns a list.
+# Get a cell from a column of complex type returns a list
instead of a vector.
col <- listCols[[colIndex]]
+colName <- dtypes[[colIndex]][[1]]
if (length(col) <= 0) {
- df[[names[colIndex]]] <- col
+ df[[colName]] <- col
} else {
- # TODO: more robust check on column of primitive types
- vec <- do.call(c, col)
- if (class(vec) != "list") {
-df[[names[colIndex]]] <- vec
+ colType <- dtypes[[colIndex]][[2]]
+ if (!is.null(PRIMITIVE_TYPES[[colType]]) && colType !=
"binary") {
+vec <- do.call(c, col)
+stopifnot (class(vec) != "list")
--- End diff --
nit: no space for func call: `stopifnot(class(vec) != "list")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-159458260 @shivaram, @felixcheung, could you take more look? another PR depends on this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-158922986 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46532/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-158922778 **[Test build #46532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46532/consoleFull)** for PR 9769 at commit [`2af83b7`](https://github.com/apache/spark/commit/2af83b7787ad7c3bb867131c9f28be3f25198277). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-158922984 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-158917368 **[Test build #46532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46532/consoleFull)** for PR 9769 at commit [`2af83b7`](https://github.com/apache/spark/commit/2af83b7787ad7c3bb867131c9f28be3f25198277). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-158916810 @felixcheung, you concern is reasonable. I refactor the code by using schema to determine if a collected column can be coerced into an atomic vector. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9769#discussion_r45419720
--- Diff: R/pkg/R/DataFrame.R ---
@@ -717,25 +717,30 @@ setMethod("collect",
# data of complex type can be held. But getting a cell
from a column
# of list type returns a list instead of a vector. So for
columns of
# non-complex type, append them as vector.
+#
+# For columns of complex type, be careful to access them.
+# Get a column of complex type returns a list.
+# Get a cell from a column of complex type returns a list
instead of a vector.
col <- listCols[[colIndex]]
if (length(col) <= 0) {
df[[names[colIndex]]] <- col
} else {
# TODO: more robust check on column of primitive types
- vec <- do.call(c, col)
- if (class(vec) != "list") {
-df[[names[colIndex]]] <- vec
+ if (!any(sapply(col, function(e) { length(e) > 1 }))) {
--- End diff --
should this be returned from org.apache.spark.sql.api.r.SQLUtils.dfToCols
for more optimal processing? this seems like potentially a lot of data to go
through
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157614027 Ok thanks for the clarification. It might take me a couple of days to get to this as the change looks a bit involved. cc @felixcheung --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157597785 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157597786 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46159/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157597709 **[Test build #46159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46159/consoleFull)** for PR 9769 at commit [`df9bca7`](https://github.com/apache/spark/commit/df9bca79bc267dea767893cd3b394a69cc1d591b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157593732 **[Test build #46159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46159/consoleFull)** for PR 9769 at commit [`df9bca7`](https://github.com/apache/spark/commit/df9bca79bc267dea767893cd3b394a69cc1d591b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157589422 @shivaram, The R raw type is intended to hold raw bytes. while int vector is to hold 32-bit integer values. The R raw type maps to Spark SQL binary type, which is internally represented in Array[Byte]. This PR solves two problems: 1. Inferring of raw type is incorrect. ``` > SparkR:::infer_type(as.raw(c(1, 2 ,3))) [1] "array" ``` This is not correct, it should be "binary". 2. Collecting a DataFrame fails if there is any column of binary type. The bug lies in the logic that determines whether a collected column can be coerced into a atomic vector or not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157585916 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157585922 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46148/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157585897 **[Test build #46148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46148/consoleFull)** for PR 9769 at commit [`07a0f33`](https://github.com/apache/spark/commit/07a0f33556a4c9ca5ac6daca90e3c3b97a1d83f5). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157584455 **[Test build #46148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46148/consoleFull)** for PR 9769 at commit [`07a0f33`](https://github.com/apache/spark/commit/07a0f33556a4c9ca5ac6daca90e3c3b97a1d83f5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157447807 Could you describe the problem a bit more ? Was it that raw vectors were being treated as lists ? Just curious how raw vectors differ from int vectors etc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157385628 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9769#issuecomment-157385631 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46086/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11781][SPARKR] SparkR has problem in in...
GitHub user sun-rui opened a pull request: https://github.com/apache/spark/pull/9769 [SPARK-11781][SPARKR] SparkR has problem in inferring type of raw type. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sun-rui/spark SPARK-11781 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9769.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9769 commit b585ec57cc28b42466689998934b0327a5cb56ac Author: Sun Rui Date: 2015-11-17T14:10:44Z [SPARK-11781][SPARKR] SparkR has problem in inferring type of raw type. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
