Gabriel39 commented on code in PR #67574:
URL: https://github.com/apache/doris/pull/67574#discussion_r3949671583
##########
be/src/format_v2/column_mapper.cpp:
##########
@@ -2811,6 +2811,14 @@ Status TableColumnMapper::localize_filters(const
std::vector<TableFilter>& table
FileScanRequestBuilder builder(file_request);
RETURN_IF_ERROR(builder.add_non_predicate_column(std::move(demoted_projection)));
}
+ // Predicate demotion can widen a nested projection after mappings were
localized. Reapply the
+ // final shape so TableReader interprets the same child ordinals that
FileReader returns.
+ for (auto& mapping : _mappings) {
+ if (mapping.file_local_id.has_value() &&
+
file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) {
+
RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request,
&mapping));
Review Comment:
Fixed in 312d7d0f21. The mapper now reconciles visible output projections
after table-format customization, rebuilds projection expressions from the
final nested layout, and drops stale file-local conjuncts when child ordinals
change. The new Parquet and ORC regression test covers a hidden equality-delete
key widening the same struct root as a visible sibling.
##########
be/src/format_v2/table/iceberg_reader.h:
##########
@@ -73,6 +73,12 @@ class IcebergTableReader : public format::TableReader {
if (!_data_reader.file_schema.empty() && has_field_ids) {
return format::TableColumnMappingMode::BY_FIELD_ID;
}
+ if (!_data_reader.file_schema.empty() &&
supports_iceberg_scan_semantics_v2(_scan_params) &&
Review Comment:
We are intentionally keeping the existing scan-semantics version for this
change. Mixed-version compatibility handling is out of scope for this bug fix
and will not be addressed in this PR.
##########
be/src/format_v2/table/iceberg_reader.h:
##########
@@ -73,6 +73,12 @@ class IcebergTableReader : public format::TableReader {
if (!_data_reader.file_schema.empty() && has_field_ids) {
return format::TableColumnMappingMode::BY_FIELD_ID;
}
+ if (!_data_reader.file_schema.empty() &&
supports_iceberg_scan_semantics_v2(_scan_params) &&
+ !schema_has_any_authoritative_name_mapping(_projected_columns)) {
Review Comment:
Fixed in 312d7d0f21. Iceberg mapping mode now consults the complete scan
schema when projected columns do not carry the authoritative name-mapping
signal. A metadata-only equality-delete regression covers both Parquet and ORC
ID-less data files.
##########
regression-test/suites/external_table_p0/iceberg/test_gen_iceberg_by_api.groovy:
##########
@@ -48,7 +48,9 @@ suite("test_gen_iceberg_by_api", "p0,external") {
try {
qt_q02 """ select count(*) from table_with_append_file where
MAN_ID is not null """
} catch (Exception e) {
- assertTrue(e.getMessage().contains("name_mapping must be set when
read missing field id data file."), e.getMessage());
+ // This fixture has no field IDs or authoritative name mapping, so
its required
+ // columns must be treated as missing instead of being matched by
their current names.
+ assertTrue(e.getMessage().contains("Missing required field:
MAN_ID"), e.getMessage());
Review Comment:
Fixed in 312d7d0f21. The case now uses the strict test/exception assertion,
and the stale success golden result has been removed. External regression now
consistently runs the default FileScannerV2 path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
