github-actions[bot] commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4003985078
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java:
##########
@@ -247,9 +249,15 @@ protected void initSchemaParams() throws UserException {
setColumnPositionMapping();
// For query, set src tuple id to -1.
params.setSrcTupleId(-1);
- // Set enable_mapping_varbinary from catalog or TVF
params.setEnableMappingVarbinary(getEnableMappingVarbinary());
params.setEnableMappingTimestampTz(getEnableMappingTimestampTz());
+ // The marker makes an omitted timezone an explicit wall-clock choice
while old FE plans
+ // remain distinguishable during a BE-first rolling upgrade.
+
params.setParquetTimestampSemanticsVersion(FileFormatUtils.PARQUET_TIMESTAMP_SEMANTICS_VERSION);
+ String hiveParquetTimeZone = getHiveParquetTimeZone();
+ if (hiveParquetTimeZone != null && !hiveParquetTimeZone.isEmpty()) {
+ params.setHiveParquetTimeZone(hiveParquetTimeZone);
Review Comment:
[P1] Apply the INT96 timezone contract to Hive writes too
This only sends the catalog timezone on scans. Hive INSERT still forces
INT96 but builds the writer from the session zone. Worse, when this property is
unset the version marker makes reads preserve the raw INT96 wall-clock carrier,
while a non-UTC insert session writes a timezone-normalized instant; for
example, inserting `2023-04-20 00:00` in Asia/Shanghai reads back as
`2023-04-19 16:00`. With a named catalog zone, a different insert session
similarly shifts the result. Transport the full named-or-wall-clock contract
through `THiveTableSink`/`VHivePartitionWriter`: use the named zone when
configured and wall-clock/UTC normalization when empty, and add Hive
INSERT/read regressions for both cases. This is distinct from the earlier fix
that only made INT96 timezone-aware.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java:
##########
@@ -103,8 +104,14 @@ public DataType atomic(Type atomic) {
return new DateType();
} else if (primitiveType.equals(PrimitiveType.DECIMALV2) ||
primitiveType.isDecimalV3Type()) {
return new DecimalType(((ScalarType) atomic).getScalarPrecision(),
((ScalarType) atomic).getScalarScale());
- } else if (primitiveType.equals(PrimitiveType.DATETIME) ||
primitiveType.equals(PrimitiveType.DATETIMEV2)) {
+ } else if (primitiveType.equals(PrimitiveType.DATETIME)) {
return new TimestampType();
+ } else if (primitiveType.equals(PrimitiveType.DATETIMEV2)) {
+ // Preserve wall-clock semantics and fractional precision in the
Paimon schema.
+ return new TimestampType(((ScalarType) atomic).getScalarScale());
+ } else if (primitiveType.equals(PrimitiveType.TIMESTAMPTZ)) {
+ // TIMESTAMPTZ represents an instant, so map it to Paimon's
local-zoned timestamp.
+ return new LocalZonedTimestampType(((ScalarType)
atomic).getScalarScale());
} else if (primitiveType.isVarbinaryType()) {
return new VarBinaryType(VarBinaryType.MAX_LENGTH);
Review Comment:
[P1] Preserve TIMESTAMPTZ instants in Paimon LTZ writes
This newly permits TIMESTAMPTZ to create a Paimon LTZ column, but
`PaimonWriteTarget` still maps every LTZ field with timestamp-TZ mapping
disabled, so `BindSink` casts a TIMESTAMPTZ source to DATETIMEV2 before
writing. That round trip is lossy at a DST fold: in America/Los_Angeles,
`08:30Z` and `09:30Z` both render as civil `01:30`, and the DATETIMEV2
serializer can map that civil value back to only one instant. The new
TIMESTAMPTZ Paimon SerDe would preserve the UTC value, but this coercion
bypasses it. Use TIMESTAMPTZ as the LTZ write-boundary type, letting DATETIMEV2
inputs cast once, and cover an `INSERT ... SELECT` containing both sides of a
fall-back fold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
