wesm commented on a change in pull request #7611:
URL: https://github.com/apache/arrow/pull/7611#discussion_r449088163
##########
File path: r/src/array_from_vector.cpp
##########
@@ -1150,6 +1150,25 @@ std::shared_ptr<arrow::DataType>
InferArrowTypeFromVector<REALSXP>(SEXP x) {
return float64();
}
+template <>
+std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<STRSXP>(SEXP x) {
+ // See how big the character vector is
+ R_xlen_t n = XLENGTH(x);
+ int64_t size = 0;
+ for (R_xlen_t i = 0; i < n; i++) {
+ SEXP string_i = STRING_ELT(x, i);
+ if (string_i != NA_STRING) {
+ size += XLENGTH(Rf_mkCharCE(Rf_translateCharUTF8(string_i), CE_UTF8));
+ }
+ if (size > 2147483646) {
+ // Exceeds 2GB capacity of utf8 type, so use large
+ return large_utf8();
+ }
+ }
+
+ return utf8();
+}
Review comment:
Got it, thanks. Let me hack on this a little bit and see what I can do
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
