[
https://issues.apache.org/jira/browse/ARROW-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555180#comment-17555180
]
Thomas Buhrmann commented on ARROW-16843:
-----------------------------------------
Or even expose the type inference itself in some way, so one could simply read
all columns as strings and then use the underlying type inference on a column
by column basis, using additional custom logic. I'm currently creating an
additional inference layer, e.g., that also infers list types from string
columns, timestamps with non-iso formats, downcasts ints to the smallest
possible type etc. (the uint64 case if the only "problem" I had so far fwiw..)
> [Python][CSV] CSV reader performs unsafe type conversion
> --------------------------------------------------------
>
> Key: ARROW-16843
> URL: https://issues.apache.org/jira/browse/ARROW-16843
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 8.0.0
> Reporter: Thomas Buhrmann
> Priority: Major
>
> Hi, I've noticed that although pa.scalar and pa.array behave correctly when
> given the largest possible (uint64) value (i.e. they fail correctly when
> trying to cast to float e.g.), the CSV reader happily converts strings
> representing uint64 values to float (see example below). Is this intended?
> Would it be possible to have a safe-conversion-only option?
> The problem is that at the moment the only safe option to read a CSV whose
> types are not known in advance is to read without any conversion (string
> only) and perform the type inference oneself.
> It would be ok if Uint64 types couldn't be inferred, as long as the
> corresponding columns aren't coerced in a destructive manner to float. I.e.,
> if they were left as string columns, one could then implement a custom
> conversion, while still benefiting from the correct and automatic conversion
> of the remaining columns.
>
> The following correctly rejects the float type for uint64 values:
> {code:java}
> import pyarrow as pa
> uint64_max = 18_446_744_073_709_551_615
> type_ = pa.uint64()
> uint64_scalar = pa.scalar(uint64_max, type=type_)
> uint64_array = pa.array([uint64_max], type=type_)
> try:
> f = pa.scalar(uint64_max, type=pa.float64())
> except Exception as exc:
> print(exc)
>
> try:
> f = pa.scalar(uint64_max // 2, type=pa.float64())
> except Exception as exc:
> print(exc) {code}
> {code:java}
> >> PyLong is too large to fit int64
> >> Integer value 9223372036854775807 is outside of the range exactly
> >> representable by a IEEE 754 double precision value
> {code}
> The CSV reader, on the other hand, doesn't infer UInt64 types (which is fine,
> as documented here
> [https://arrow.apache.org/docs/cpp/csv.html#data-types),|https://arrow.apache.org/docs/cpp/csv.html#data-types)]
> but does coerce values to float which shouldn't be coercable according to
> above examples:
> {code:java}
> import io
> csv = "int64,uint64\n0,0\n4294967295,18446744073709551615"
> tbl = pa.csv.read_csv(io.BytesIO(csv.encode("utf-8")))
> print(tbl.schema)
> print(tbl.column("uint64")[1] == uint64_scalar)
> print(tbl.column("uint64")[1].cast(pa.uint64())) {code}
> {code:java}
> int64: int64
> uint64: double
> False
> 0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
