morningman commented on code in PR #69697:
URL: https://github.com/apache/doris/pull/69697#discussion_r4488438594
##########
be/src/core/data_type_serde/data_type_datetimev2_serde.cpp:
##########
@@ -723,7 +723,22 @@ Status
DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
--seconds;
remainder += divisor;
}
- v.from_unixtime(seconds, real_ctz);
+ // from_unixtime() narrows the cctz year to uint16_t without a
check, so an instant
+ // whose civil year in real_ctz is outside 0000..9999 would read
as another value:
+ // timestamp(us) 2005980768000000000 (65537-01-01) as 0001-01-01,
and the years
+ // 10000..65535 as dates DATETIMEV2 cannot hold. Check the civil
year before packing,
+ // as orc_timestamp_to_datetime() does for ORC.
+ const auto civil = cctz::convert(
+ cctz::time_point<cctz::seconds>(cctz::seconds(seconds)),
real_ctz);
+ if (civil.year() < 0 || civil.year() > 9999) {
Review Comment:
Follow-up on the STRUCT and LIST cases promised above. This PR was merged on
2026-11-03, before #69581, so the cases went into #69581 instead, which was
merged on 2026-11-06 as 0d1ae9ee064:
`DataTypeDateTimeV2SerDeTest.IgnoresArrowTimestampsHiddenByNullParents` in
`be/test/core/data_type_serde/data_type_serde_datetime_v2_test.cpp`. For both
DATETIMEV2 and TIMESTAMPTZ, it hides 65537-01-01 (2005980768000000000) under
the NULL row of a LIST, a FIXED_SIZE_LIST, a MAP value and a STRUCT. The NULL
row reads as NULL and the visible rows keep their values, while the same value
in a visible row still fails with the range error. Returning the child unmasked
from `mask_arrow_children_hidden_by_null_parents` fails the test, and dropping
only the STRUCT mask fails it at the STRUCT case.
A correction to my reply above: master did carry the regression between the
two merges, from 2026-11-03 to 2026-11-06. In that window, a batch whose NULL
list, map or struct row hid an out-of-range timestamp failed with
DataQualityError instead of being read. It raised an error and did not return
wrong values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
