nankeChen75 opened a new pull request, #12616:
URL: https://github.com/apache/seatunnel/pull/12616
### Purpose of this pull request
Fixes #9788.
Apache Arrow represents `TIMESTAMPMICROTZ` values with
`TimeStampMicroTZVector`, whose values are epoch-based microseconds.
SeaTunnel did not have a converter for this Arrow type, so the value fell
back to the default conversion path and was interpreted as epoch milliseconds.
This produced timestamps with abnormally large years, for example
`+57626-09-11T22:15:23.456`. When such values were written back to Doris
`DATETIMEV2`, Doris stored them as `NULL`.
This PR:
- Adds `TimeStampMicroTZConverter` for Arrow `TIMESTAMPMICROTZ`.
- Converts epoch microseconds without losing precision.
- Converts the resulting instant to a `LocalDateTime` using the system time
zone, consistently with the existing timezone-aware timestamp converters.
- Adds regression coverage to `ArrowToSeatunnelRowReaderTest`.
### Does this PR introduce _any_ user-facing change?
Yes.
Previously, reading a Doris `DATETIMEV2` column represented by Arrow as
`TIMESTAMPMICROTZ` could produce an invalid far-future timestamp because the
microsecond value was interpreted as milliseconds.
For example:
```text
+57626-09-11T22:15:23.456
```
In a Doris-to-Doris job, these invalid timestamps could subsequently be
written as `NULL`.
After this change, `TIMESTAMPMICROTZ` values are converted using the correct
microsecond unit and retain their microsecond precision.
No configuration or public API changes are introduced.
### How was this patch tested?
Added regression coverage to:
```text
ArrowToSeatunnelRowReaderTest
```
The test creates a `TimeStampMicroTZVector`, writes an epoch-microsecond
value with six-digit fractional precision, reads it through
`ArrowToSeatunnelRowReader`, and verifies the resulting `LocalDateTime`.
The targeted test was run with JDK 8:
```bash
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin:$PATH \
./mvnw \
-pl seatunnel-connectors-v2/connector-common \
-Dtest=ArrowToSeatunnelRowReaderTest \
test
```
Result:
```text
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
The project was also compiled and verified without running tests:
```bash
./mvnw -q -DskipTests verify
```
The issue was reproduced before the fix with:
1. A Doris source-to-Console job, which showed that the corrupted far-future
timestamp was produced before the sink.
2. A Doris source-to-Doris job, in which both source rows were read and
written successfully at the job-metrics level, but the target `DATETIMEV2`
values became `NULL`.
### Check list
* [x] No new JAR binary package is introduced.
* [x] No documentation update is required because this change does not add
or modify configuration, public APIs, or connector usage.
* [x] No incompatible change is introduced.
* [x] The new source file contains the Apache License header.
* [x] The change includes a regression test.
* [x] The connector registration checklist is not applicable because this PR
fixes shared Arrow conversion logic for existing connectors and does not add a
new connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
