Gabriel39 commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4015257651
##########
be/src/exec/operator/file_scan_operator.cpp:
##########
@@ -147,9 +148,23 @@ bool FileScanLocalState::should_use_file_scanner_v2(const
TQueryOptions& query_o
const bool is_transactional_hive =
scan_params.__isset.table_format_params &&
scan_params.table_format_params.table_format_type ==
"transactional_hive";
- return query_options.__isset.enable_file_scanner_v2 &&
query_options.enable_file_scanner_v2 &&
- !is_load && scan_params.format_type !=
TFileFormatType::FORMAT_ES_HTTP &&
- !is_transactional_hive;
+ const bool is_paimon_native_parquet = scan_params.format_type ==
TFileFormatType::FORMAT_JNI &&
+
scan_params.__isset.contains_native_parquet &&
+ scan_params.contains_native_parquet;
+ const bool requires_parquet_timestamp_contract =
Review Comment:
Fixed in 78ded3952e. Versioned Iceberg scans requiring the Parquet timestamp
contract now select V2 independently of the current default write format. This
also covers deferred/remote splits without eagerly enumerating manifests. Added
regression coverage for an ORC-default scan before range delivery and for
row-ID phase-two reads of a retained Parquet range; both failed before the
routing fix and now pass.
##########
be/src/core/data_type_serde/data_type_serde.h:
##########
@@ -498,6 +499,23 @@ class DataTypeSerDe {
virtual Status write_column_to_arrow(const IColumn& column, const NullMap*
null_map,
arrow::ArrayBuilder* array_builder,
int64_t start,
int64_t end, const cctz::time_zone&
ctz) const = 0;
+ // Most scalar types deliberately share their physical Arrow encoding
across these protocols.
+ // Target-specific SerDes override the corresponding method; callers never
retry another
+ // protocol method after an error.
+ virtual Status write_column_to_paimon(const std::shared_ptr<const
IDataType>&,
Review Comment:
Renamed to write_column_to_paimon_arrow in 78ded3952e, including all
overrides, nested SerDe recursion, and converter call sites. Verified that the
rename does not change serialization logic; the affected production objects
compile and the Arrow converter/SerDe tests pass.
##########
be/src/core/data_type_serde/data_type_serde.h:
##########
@@ -498,6 +499,23 @@ class DataTypeSerDe {
virtual Status write_column_to_arrow(const IColumn& column, const NullMap*
null_map,
arrow::ArrayBuilder* array_builder,
int64_t start,
int64_t end, const cctz::time_zone&
ctz) const = 0;
+ // Most scalar types deliberately share their physical Arrow encoding
across these protocols.
+ // Target-specific SerDes override the corresponding method; callers never
retry another
+ // protocol method after an error.
+ virtual Status write_column_to_paimon(const std::shared_ptr<const
IDataType>&,
+ const IColumn& column, const
NullMap* null_map,
+ const std::shared_ptr<arrow::Field>&,
+ arrow::ArrayBuilder* array_builder,
int64_t start,
+ int64_t end, const cctz::time_zone&
ctz) const {
+ return write_column_to_arrow(column, null_map, array_builder, start,
end, ctz);
+ }
+ virtual Status write_column_to_iceberg(const std::shared_ptr<const
IDataType>&,
Review Comment:
Renamed to write_column_to_iceberg_arrow in 78ded3952e, including all
overrides, nested SerDe recursion, and converter call sites. Verified that the
rename does not change serialization logic; the affected production objects
compile and the Arrow converter/SerDe tests pass.
##########
be/src/format_v2/table_reader.cpp:
##########
@@ -1636,6 +1641,8 @@ Status
TableReader::create_file_reader(std::unique_ptr<FileReader>* reader) {
const bool enable_mapping_varbinary = _scan_params != nullptr &&
_scan_params->__isset.enable_mapping_varbinary &&
_scan_params->enable_mapping_varbinary;
+ const std::optional<std::string> hive_parquet_time_zone =
Review Comment:
Fixed in 78ded3952e. The Parquet condition-cache digest now includes the
normalized INT96 timezone override. Versioned omission and explicit empty share
the wall-clock identity, named zones are isolated, and legacy absence retains
the existing session-based seed. The predicate seed is not mutated across
splits. Added a real INT96 regression that verifies the physical file type and
unfiltered timezone values: without the fix, a cached all-false granule loses
all 2048 matching rows under Asia/Shanghai; with the fix, all 2048 rows are
returned and repeated scans hit the correct cache. The new regressions and the
complete selected set of 239 BE tests pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
