Gabriel39 commented on code in PR #65784:
URL: https://github.com/apache/doris/pull/65784#discussion_r3611966085
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -549,12 +524,24 @@ Map<Integer, String>
getBase64EncodedInitialDefaultsForScan() throws UserExcepti
// schema that produced source.getTargetTable().getColumns() to
keep defaults aligned.
return
IcebergUtils.getBase64EncodedInitialDefaults(icebergTable.schema());
}
+ IcebergTableQueryInfo selectedSnapshot = getSpecifiedSnapshot();
+ if (selectedSnapshot == null) {
+ Optional<MvccSnapshot> mvccSnapshot =
MvccUtil.getSnapshotFromContext(source.getTargetTable());
+ Schema scanSchema = icebergTable.schema();
+ if (mvccSnapshot.isPresent() && mvccSnapshot.get() instanceof
IcebergMvccSnapshot) {
+ long schemaId = ((IcebergMvccSnapshot) mvccSnapshot.get())
+ .getSnapshotCacheValue().getSnapshot().getSchemaId();
+ scanSchema =
icebergTable.schemas().get(Math.toIntExact(schemaId));
+ }
+ // The statement snapshot produced the target columns; cache
invalidation must not
+ // let initial-default metadata advance to a different schema
during the same scan.
+ return IcebergUtils.getBase64EncodedInitialDefaults(
+ Preconditions.checkNotNull(scanSchema, "Schema for Iceberg
scan is null"));
+ }
TableScan tableScan = createTableScan();
Snapshot snapshot = tableScan.snapshot();
- // TableScan.schema() starts from the table's current schema even for
useSnapshot/useRef.
- // Resolve the selected snapshot's schema id explicitly so this
metadata describes the same
- // snapshot as source.getTargetTable().getColumns(). Otherwise a later
type change can make
- // BE decode a historical non-binary default as Base64, or fail to
decode a binary default.
+ // Explicit time travel and ref scans expose the selected snapshot
schema rather than the
Review Comment:
Fixed in c4f800c8511. Default markers now use the statement-pinned Iceberg
MVCC schema, while retaining the resolved-selection fallback when no MVCC
snapshot is available. Added a schema-only old-branch binary-default regression
test.
##########
be/src/format/table/table_schema_change_helper.cpp:
##########
@@ -825,27 +958,27 @@ Status
TableSchemaChangeHelper::BuildTableInfoUtil::by_orc_field_id_with_name_ma
auto struct_node = std::make_shared<TableSchemaChangeHelper::StructNode>();
std::map<int32_t, size_t> file_column_id_idx_map;
- bool all_have_field_id = true;
+ // Iceberg ORC builds projection by ID for the entire subtree. An ID-less
wrapper containing
+ // an ID-bearing child is therefore absent rather than rebound by its
physical name.
+ bool has_field_id = orc_subtree_has_field_id(orc_root,
field_id_attribute_key);
Review Comment:
Fixed in c4f800c8511. The file-wide ORC ID-projection decision is now
threaded through struct, array, and map recursion, so nested ID-less structs
cannot re-enable name matching. Added an array/map regression test.
##########
be/src/format/table/table_schema_change_helper.cpp:
##########
@@ -604,6 +726,13 @@ Status
TableSchemaChangeHelper::BuildTableInfoUtil::by_parquet_field_id_with_nam
Status
TableSchemaChangeHelper::BuildTableInfoUtil::by_parquet_field_id_with_name_mapping(
const schema::external::TField& table_schema, const FieldSchema&
parquet_field,
std::shared_ptr<TableSchemaChangeHelper::Node>& node) {
+ return by_parquet_field_id_with_name_mapping(table_schema, parquet_field,
node,
Review Comment:
Fixed in c4f800c8511. The V1 position-delete reader now maps row against the
complete Parquet/ORC delete-file schema, so file_path/pos IDs keep row in
file-wide ID mode. Added regressions for both formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
