[
https://issues.apache.org/jira/browse/ARROW-11067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256605#comment-17256605
]
Neal Richardson commented on ARROW-11067:
-----------------------------------------
I can reproduce this on macOS 10.14 / R 3.6, as well as on the new M1 mac with
11.0 / R 4.0:
{code}
tab <- read_csv_arrow("~/Downloads/demo_data.csv", as_data_frame=FALSE)
sum(is.na(tab$json_string))
## Scalar
## 10
{code}
But on my macOS 10.14, I also see what Weston found in Python: it works.
And--if I call pyarrow through R with reticulate, with or without arrow
involved, it also works, no missing values:
{code}
library(reticulate)
csv <- import("pyarrow.csv")
tab <- csv$read_csv("/Users/enpiar/Downloads/demo_data.csv")
strs <- tab$column("json_string")$to_pylist()
nchar(strs)
## [1] 38660 37627 34127 45107 127507 59150 34426 41492 39564 106966
## [11] 880 7882 216 734 407 31111 383 217 887 357
{code}
{code}
library(arrow)
library(reticulate)
csv <- import("pyarrow.csv")
tab <- csv$read_csv("/Users/enpiar/Downloads/demo_data.csv")
sum(is.na(tab$json_string))
## Scalar
## 0
{code}
Perhaps this is pure coincidence, but I see some commits on r-devel in recent
days that mention some kind of odd behavior with strings on macOS, e.g.
https://github.com/r-devel/r-svn/commit/1c01b0f8fc817b77a1b6d6091c20168090f2ac15.
So maybe there is some locale setting that R uses, which affects the
environment that the Arrow C++ library runs in (in process) but doesn't affect
a Python subprocess, that causes string parsing errors in Arrow?
> [R] read_csv_arrow silently fails to read some strings and returns nulls
> ------------------------------------------------------------------------
>
> Key: ARROW-11067
> URL: https://issues.apache.org/jira/browse/ARROW-11067
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: John Sheffield
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: arrow_explanation.png, arrow_failure_cases.csv,
> arrow_failure_cases.csv, arrowbug1.png, arrowbug1.png, demo_data.csv
>
>
> A sample file is attached, showing 10 rows each of strings with consistent
> failures (false_na = TRUE) and consistent successes (false_na = FALSE). The
> strings are in the column `json_string` – if relevant, they are geojsons with
> min nchar of 33,229 and max nchar of 202,515.
> When I read this sample file with other R CSV readers (readr and data.table
> shown), the files are imported correctly and there are no NAs in the
> json_string column.
> When I read with arrow::read_csv_arrow, 50% of the sample json_string column
> end up as NAs. as_data_frame TRUE or FALSE does not change the behavior, so
> this might not be limited to the R interface, but I can't help debug much
> further upstream.
>
>
> {code:java}
> aaa1 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = TRUE)
> aaa2 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = FALSE)
> bbb <- data.table::fread("demo_data.csv")
> ccc <- readr::read_csv("demo_data.csv")
> mean(is.na(aaa1$json_string)) # 0.5
> mean(is.na(aaa2$column(1))) # Scalar 0.5
> mean(is.na(bbb$json_string)) # 0
> mean(is.na(ccc$json_string)) # 0{code}
>
>
> * arrow 2.0 (latest CRAN)
> * readr 1.4.0
> * data.table 1.13.2
> * R version 4.0.1 (2020-06-06)
> * MacOS Catalina 10.15.7 / x86_64-apple-darwin17.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
