jorisvandenbossche commented on a change in pull request #11358:
URL: https://github.com/apache/arrow/pull/11358#discussion_r733612684
##########
File path: docs/source/cpp/csv.rst
##########
@@ -190,6 +190,70 @@ dictionary-encoded string-like array. It switches to a
plain string-like
array when the threshold in :member:`ConvertOptions::auto_dict_max_cardinality`
is reached.
+Timestamp inference/parsing
+---------------------------
+
+If type inference is enabled, the CSV reader first tries to interpret
+string-like columns as timestamps. If all rows have some zone offset
+(e.g. ``Z`` or ``+0100``), even if the offsets are inconsistent, then the
+inferred type will be UTC timestamp. If no rows have a zone offset, then the
+inferred type will be timestamp without timezone. A mix of rows with/without
+offsets will result in a string column.
+
+If the type is explicitly specified as a timestamp without timezone ("naive"),
+then the reader will error on values with zone offsets in that column. Else, if
+the type is timestamp with timezone, the column values must either all have
+zone offsets or all lack zone offsets. In the former case, values are
+unambiguous, since each row specifies a precise time in UTC, but in the latter
+case, Arrow will currently interpret the timestamps as specifying values in UTC
+(i.e. as if they had the zone offset "Z" or "+0000"), *not* as values in the
+local time of the timezone.
Review comment:
> So this means that
`read_csv(...).column("start_time").cast(pa.timestamp('s', 'Europe/Brussels'))`
would give a different answer than `read_csv(..., types={'start_time':
pa.timestamp('s', 'Europe/Brussels')}).column("start_time")`.
I think we should ensure those two are equivalent. _If_ we interpret native
strings as local time when specifying a timezone-aware type in the csv parsing,
I think casting should have the same behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
