Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
Thanks for the reminder. I may have forgotten to mention that I am the
reporter of this JIRA bug. My JIRA ID is also titicaca. Thank you!
---
If your project is set up for it, you can reply
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
Yes. The JIRA id is SPARK-19342. Thank you for the help and advices :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
Yes, collect on timestamp was getting `c("POSIXct", "POSIXt")`. But when NA
exists at the top of the timetamp column, it was getting `numeric` as I
described in the PR descri
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
Thanks. I tried to fix the method `coltypes` for the modification of the
timestamp, and it can pass all the tests now.
---
If your project is set up for it, you can reply to this email and have
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
I tried to modify the PRIMITIVE_TYPES for timestamp, but it had a side
effect on coltypes method.
In test_sparkSQL.R#2262, `expect_equal(coltypes(DF), c("integer",
"log
Github user titicaca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16689#discussion_r98918704
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1138,6 +1138,11 @@ setMethod("collect",
if (!is.null(PRIMITIVE_TYPES[[colType]])
Github user titicaca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16689#discussion_r98612545
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1138,6 +1138,11 @@ setMethod("collect",
if (!is.null(PRIMITIVE_TYPES[[colType]])
Github user titicaca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16689#discussion_r98611766
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1138,6 +1138,11 @@ setMethod("collect",
if (!is.null(PRIMITIVE_TYPES[[colType]])
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
I have modified the codes and tests, including the existed tests
@test_sparkSQL.R#1280 and @test_sparkSQL.R#1282.
Like in local R, now NA column of the SparkDataFrame will also
Github user titicaca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16689#discussion_r97714703
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1138,6 +1138,9 @@ setMethod("collect",
if (!is.null(PRIMITIVE_TYPES[[colType]])
Github user titicaca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16689#discussion_r97712469
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1138,6 +1138,9 @@ setMethod("collect",
if (!is.null(PRIMITIVE_TYPES[[colType]])
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
Sorry for the late reply. I figured out that the tests failed because if a
vector is with only NAs, the type is logical, therefore we cannot cast the type
in that case. I have updated the codes
Github user titicaca commented on the issue:
https://github.com/apache/spark/pull/16689
Sure. Shall I add the tests in pkg/inst/tests/testthat/test_sparkSQL.R?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user titicaca opened a pull request:
https://github.com/apache/spark/pull/16689
SPARK-19342 bug fixed in collect method for collecting timestamp column
## What changes were proposed in this pull request?
Fix a bug in collect method for collecting timestamp column
14 matches
Mail list logo
