"R version 3.4.0 (2017-04-21)" on "x86_64-w64-mingw32" platform
I am using CSVs and other text tables, and text in general (including regular expressions), on Windows 10. For me, that means dealing with Windows-1252 and UTF-8 encoding, with UTF-16 and UTF-32 as helpful curiosities. Something as simple as iconv ("\n", to = "UTF-16") causes an error, due to an embedded nul. Then there is write.csv (or write.table) with its fileEncoding parameter: not working correctly for UTF-16 and UTF-32. Of course, developers are aware of this, for example [Rd] iconv to UTF-16 encoding produces error due to embedded nulls (write.table with fileEncoding param) https://stat.ethz.ch/pipermail/r-devel/2016-February/072323.html iconv to UTF-16 encoding produces error due to embedded nulls (write.table with fileEncoding param) http://r.789695.n4.nabble.com/iconv-to-UTF-16-encoding-produces-error-due-to -embedded-nulls-write-table-with-fileEncoding-param-td4717481.html ---------------------------------------------------------------------------- ------------------------ Focussing on write.csv and UTF-16LE and UTF-16BE, it seems that a nul character is omitted in each <CarriageReturn><LineFeed> pair. TEST SCRIPT ---------------------------------------------------------------------------- ------------------------ remove (list = objects()) print (sessionInfo()) cat ("---------------------------------\n\n") LE <- data.frame ( want = c ("0d,00", "0a,00"), got = c ("0d ", "0a,00") ) BE <- data.frame ( want = c ("00,0d", "00,0a"), got = c ("00,0d", " 0a") ) write.csv (LE, "R_LE.csv", fileEncoding = "UTF-16LE", row.names = FALSE) write.csv (BE, "R_BE.csv", fileEncoding = "UTF-16BE", row.names = FALSE) print (readBin ("R_LE.csv", "raw", 1000)) print (LE) cat ("\n") print (readBin ("R_BE.csv", "raw", 1000)) print (BE) cat ("\n") try (iconv ("\n", to = "UTF-8")) try (iconv ("\n", to = "UTF-16LE")) try (iconv ("\n", to = "UTF-16BE")) try (iconv ("\n", to = "UTF-16")) try (iconv ("\n", to = "UTF-32LE")) try (iconv ("\n", to = "UTF-32BE")) try (iconv ("\n", to = "UTF-32")) ---------------------------------------------------------------------------- ------------------------ TEST SCRIPT OUTPUT > source ("bug_encoding.R") R version 3.4.0 (2017-04-21) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 14393) Matrix products: default locale: [1] LC_COLLATE=English_Australia.1252 LC_CTYPE=English_Australia.1252 [3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C [5] LC_TIME=English_Australia.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] compiler_3.4.0 --------------------------------- [1] 22 00 77 00 61 00 6e 00 74 00 22 00 2c 00 22 00 67 00 6f 00 74 00 22 00 0d [26] 0a 00 22 00 30 00 64 00 2c 00 30 00 30 00 22 00 2c 00 22 00 30 00 64 00 20 [51] 00 20 00 20 00 22 00 0d 0a 00 22 00 30 00 61 00 2c 00 30 00 30 00 22 00 2c [76] 00 22 00 30 00 61 00 2c 00 30 00 30 00 22 00 0d 0a 00 want got 1 0d,00 0d 2 0a,00 0a,00 [1] 00 22 00 77 00 61 00 6e 00 74 00 22 00 2c 00 22 00 67 00 6f 00 74 00 22 00 [26] 0d 0a 00 22 00 30 00 30 00 2c 00 30 00 64 00 22 00 2c 00 22 00 30 00 30 00 [51] 2c 00 30 00 64 00 22 00 0d 0a 00 22 00 30 00 30 00 2c 00 30 00 61 00 22 00 [76] 2c 00 22 00 20 00 20 00 20 00 30 00 61 00 22 00 0d 0a want got 1 00,0d 00,0d 2 00,0a 0a Error in iconv("\n", to = "UTF-16LE") : embedded nul in string: '\n\0' Error in iconv("\n", to = "UTF-16BE") : embedded nul in string: '\0\n' Error in iconv("\n", to = "UTF-16") : embedded nul in string: 'þÿ\0\n' Error in iconv("\n", to = "UTF-32LE") : embedded nul in string: '\n\0\0\0' Error in iconv("\n", to = "UTF-32BE") : embedded nul in string: '\0\0\0\n' Error in iconv("\n", to = "UTF-32") : embedded nul in string: '\0\0þÿ\0\0\0\n' > ---------------------------------------------------------------------------- ------------------------ Cheers -- Jack Kelley ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel