Gabriel39 commented on code in PR #65446:
URL: https://github.com/apache/doris/pull/65446#discussion_r3709964169
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java:
##########
@@ -120,6 +120,11 @@ public HudiScanPlanProvider(Map<String, String>
properties, ConnectorContext con
this.context = context;
}
+ @Override
+ public boolean usesHiveParquetInt96TimeZone() {
+ return true;
Review Comment:
Fixed in 4af3a54491 by keeping Hudi outside the Hive INT96 compatibility
policy. Hudi native Parquet now explicitly uses the SQL session timezone, JNI
continues to use the same session timezone, and the FE provider no longer
propagates to Hudi. Native and JNI coverage uses different catalog and session
zones.
##########
be/src/format_v2/jni/hudi_jni_reader.cpp:
##########
@@ -93,6 +93,10 @@ Status
HudiJniReader::build_scanner_params(std::map<std::string, std::string>* p
(*params)["instant_time"] = hudi_params.instant_time;
(*params)["serde"] = hudi_params.serde;
(*params)["input_format"] = hudi_params.input_format;
+ (*params)["time_zone"] = _scan_params->__isset.hive_parquet_time_zone &&
Review Comment:
Fixed in 4af3a54491 by removing the Hudi-specific INT96 override entirely.
JNI uses one SQL session timezone for both and epoch-backed , matching Hudi's
pre-existing contract and the native path.
##########
be/src/format_v2/column_mapper.cpp:
##########
@@ -1803,6 +1842,7 @@ static Status add_scan_column(FileScanRequest*
file_request, ColumnMapping* mapp
// merge_filter_projection() adds `s -> b`, so the predicate column
reads both children.
RETURN_IF_ERROR(merge_filter_projection(filter_projections,
&projection));
}
+ attach_timestamp_semantics(*mapping, &projection);
Review Comment:
Fixed in 4af3a54491. Schema-fallback child projections now copy from the
history-annotated file schema, including filter-only nested children. pins the
unannotated INT96 mapper path, and now projects one struct child while
filtering on its LTZ sibling.
##########
fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiColumnValue.java:
##########
@@ -133,7 +135,7 @@ public LocalDateTime getDateTime() {
return ((Timestamp) fieldData).toLocalDateTime();
} else if (fieldData instanceof TimestampWritableV2) {
return
LocalDateTime.ofInstant(Instant.ofEpochSecond((((TimestampObjectInspector)
fieldInspector)
- .getPrimitiveJavaObject(fieldData)).toEpochSecond()),
zoneId);
+ .getPrimitiveJavaObject(fieldData)).toEpochSecond()),
int96ZoneId);
Review Comment:
Fixed in 4af3a54491. The format-agnostic JNI decoder no longer receives or
applies an INT96 override, so Hudi ORC, Parquet INT96, and logical INT64
timestamps all retain the existing SQL session-timezone behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
