Gabriel39 commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4004286499
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java:
##########
@@ -247,9 +249,15 @@ protected void initSchemaParams() throws UserException {
setColumnPositionMapping();
// For query, set src tuple id to -1.
params.setSrcTupleId(-1);
- // Set enable_mapping_varbinary from catalog or TVF
params.setEnableMappingVarbinary(getEnableMappingVarbinary());
params.setEnableMappingTimestampTz(getEnableMappingTimestampTz());
+ // The marker makes an omitted timezone an explicit wall-clock choice
while old FE plans
+ // remain distinguishable during a BE-first rolling upgrade.
+
params.setParquetTimestampSemanticsVersion(FileFormatUtils.PARQUET_TIMESTAMP_SEMANTICS_VERSION);
+ String hiveParquetTimeZone = getHiveParquetTimeZone();
+ if (hiveParquetTimeZone != null && !hiveParquetTimeZone.isEmpty()) {
+ params.setHiveParquetTimeZone(hiveParquetTimeZone);
Review Comment:
Fixed in d652b1a4cb. THiveTableSink now carries the optional catalog INT96
timezone contract: an explicit empty value selects UTC normalization for
wall-clock values, a named value overrides the insert session, and an absent
field retains the legacy behavior for older FE requests. VHivePartitionWriter
passes this to VParquetTransformer, which keeps schema and value conversion on
the same per-writer timezone without mutating RuntimeState.
The new physical Parquet writer test reproduced the 8-hour shift before the
fix and passes afterwards, including named zones, a fixed offset, and the
absent-field compatibility case. Added Hive 2/3 INSERT/read regressions across
session zones with scalar and nested timestamps. Local verification: 68 FE
tests, 30 BE tests, FE Checkstyle, clang-format 16, and ASAN/UBSAN compilation
of both modified writer translation units passed. The external integration
suites still require CI; run buildall has been requested.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java:
##########
@@ -103,8 +104,14 @@ public DataType atomic(Type atomic) {
return new DateType();
} else if (primitiveType.equals(PrimitiveType.DECIMALV2) ||
primitiveType.isDecimalV3Type()) {
return new DecimalType(((ScalarType) atomic).getScalarPrecision(),
((ScalarType) atomic).getScalarScale());
- } else if (primitiveType.equals(PrimitiveType.DATETIME) ||
primitiveType.equals(PrimitiveType.DATETIMEV2)) {
+ } else if (primitiveType.equals(PrimitiveType.DATETIME)) {
return new TimestampType();
+ } else if (primitiveType.equals(PrimitiveType.DATETIMEV2)) {
+ // Preserve wall-clock semantics and fractional precision in the
Paimon schema.
+ return new TimestampType(((ScalarType) atomic).getScalarScale());
+ } else if (primitiveType.equals(PrimitiveType.TIMESTAMPTZ)) {
+ // TIMESTAMPTZ represents an instant, so map it to Paimon's
local-zoned timestamp.
+ return new LocalZonedTimestampType(((ScalarType)
atomic).getScalarScale());
} else if (primitiveType.isVarbinaryType()) {
return new VarBinaryType(VarBinaryType.MAX_LENGTH);
Review Comment:
Fixed in d652b1a4cb. PaimonWriteTarget now uses TIMESTAMPTZ for LTZ fields
recursively, independent of the catalog read-mapping setting, so TIMESTAMPTZ
inputs no longer pass through DATETIMEV2. Static LTZ partition formatting also
consumes the UTC literal directly before converting to the SDK timezone.
Added FE write-boundary and static-partition tests, a BE serializer test
preserving both UTC instants in the Los Angeles fall-back fold, and an INSERT
... SELECT regression checking both native and JNI reads. The relevant local
FE/BE tests pass; the external integration regression is added for CI and has
not been run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
