Gabriel39 commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4011803933
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java:
##########
@@ -487,7 +488,10 @@ private void
setStaticPartitionToContext(UnboundIcebergTableSink<?> sink,
for (Map.Entry<String, Expression> entry :
staticPartitions.entrySet()) {
Expression expr = entry.getValue();
if (expr instanceof Literal) {
- staticPartitionValues.put(entry.getKey(), ((Literal)
expr).getStringValue());
+ // Binary literals expose bare hex, while BE partition
parsing and FE commit
+ // conversion require an explicit byte representation,
distinct from text.
+ String value = ((Literal) expr).getStringValue();
+ staticPartitionValues.put(entry.getKey(), expr instanceof
VarBinaryLiteral ? "0x" + value : value);
Review Comment:
Fixed in 92e7b3c40a. SQL NULL now remains null in the FE overwrite context,
and a separate optional `static_partition_null_keys` set preserves it across
the string-valued Thrift map. BE handles this state before UUID/hex decoding
and creates NULL constants explicitly during hybrid dispatch. Non-null binary
values use strict decoding instead of nullable parsing that could silently
produce empty bytes. Added FE literal/wire/overwrite-filter tests and BE
full-static/hybrid dispatch tests for BINARY, FIXED, and UUID, plus empty-byte
and malformed-value checks. The original BE failure was reproduced before the
fix. All 270 focused FE and 161 focused BE tests now pass. The Doris/Spark
regression suite was extended; its end-to-end execution remains pending CI.
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -384,6 +384,8 @@ struct THiveTableSink {
10: optional bool overwrite
11: optional THiveSerDeProperties serde_properties
12: optional list<Types.TNetworkAddress> broker_addresses;
+ // Absent: legacy session timezone; empty: wall-clock INT96; otherwise:
named catalog timezone.
+ 13: optional string hive_parquet_time_zone
Review Comment:
Fixed in 92e7b3c40a. Hive sink timezone now uses field 14, leaving field 13
reserved for the existing master Azure multipart bool. Rechecked the current
master schema before assigning the new IDs. Added independent Binary and
Compact protocol tests containing field 13 as BOOL and field 14 as STRING,
covering absent, empty, UTC, and named timezones and verifying the serialized
field IDs. The test failed with the previous field-13 schema and passes with
the fix.
##########
be/src/exec/operator/file_scan_operator.cpp:
##########
@@ -147,9 +148,22 @@ bool FileScanLocalState::should_use_file_scanner_v2(const
TQueryOptions& query_o
const bool is_transactional_hive =
scan_params.__isset.table_format_params &&
scan_params.table_format_params.table_format_type ==
"transactional_hive";
- return query_options.__isset.enable_file_scanner_v2 &&
query_options.enable_file_scanner_v2 &&
- !is_load && scan_params.format_type !=
TFileFormatType::FORMAT_ES_HTTP &&
- !is_transactional_hive;
+ const bool is_paimon_native_scan = scan_params.format_type ==
TFileFormatType::FORMAT_JNI &&
Review Comment:
Fixed in 92e7b3c40a. FE now transports `contains_native_parquet`, set only
for actual native Parquet ranges; neither native ORC history schemas nor
logical JNI splits set it. BE uses that signal instead of the history-schema
proxy. Tests cover ORC-only, JNI-only, native Parquet, and preserving the
signal when a later range is ORC, plus BE routing with scanner V2 explicitly
disabled. The original ORC routing assertion failed before the fix and now
passes. PR documentation now distinguishes native Parquet routing and
scanner-V2 ORC behavior.
##########
be/src/format_v2/parquet/parquet_timestamp_semantics.h:
##########
@@ -0,0 +1,40 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string>
+
+#include "gen_cpp/PlanNodes_types.h"
+
+namespace doris::format::parquet {
+
+inline constexpr int32_t PARQUET_TIMESTAMP_SEMANTICS_VERSION_1 = 1;
+
+inline std::optional<std::string> get_int96_timezone_override(const
TFileScanRangeParams* params) {
+ // Only the marker can distinguish a new FE's explicit wall-clock
semantics from an old FE
+ // plan whose missing timezone field must keep legacy session-timezone
decoding.
+ if (params == nullptr ||
!params->__isset.parquet_timestamp_semantics_version ||
Review Comment:
Fixed in 92e7b3c40a. An explicit field 36 now takes precedence even without
the version marker, including an explicitly empty timezone. Legacy session
fallback is retained only when neither an explicit timezone nor the new
timestamp contract is present. Scanner selection also honors explicit field 36,
so disabling V2 cannot bypass the override. Replaced the incorrect marker-only
test with an independently encoded intermediate-FE Thrift message; added
absent/empty/named timezone and version-zero checks. The new BE regression
failed before the fix and now passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
