github-actions[bot] commented on code in PR #69536:
URL: https://github.com/apache/doris/pull/69536#discussion_r4367404920
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:
##########
@@ -2327,8 +2343,73 @@ private static List<Schema> metadataSchemaHistory(Table
table) {
return new ArrayList<>(table.schemas().values());
}
- private static void addHistoricalEqualityFields(
- List<NestedField> fields, Set<Integer> missingFieldIds, Schema
historicalSchema) {
+ /**
+ * A writer commits with the schema that is current at commit, so a key
dropped before its delete commits loses
+ * the schema-only changes made before the drop from every snapshot. Field
IDs are never reused and a field
+ * only relaxes compatibly, by promotion or from required to optional, so
each field of the selected definition
+ * takes the widest promotion of its type and the optionality it has in
any retained schema; neither depends on
+ * the order of the schemas. A field required in every retained schema
stays required, and a promoted field
+ * keeps its defaults in the promoted type ({@link #promotedDefault}).
+ */
+ private static List<NestedField> withRetainedRelaxations(List<NestedField>
fields, List<Schema> schemas) {
+ List<NestedField> relaxed = new ArrayList<>(fields.size());
+ for (NestedField field : fields) {
+ Type type = field.type();
+ boolean optional = field.isOptional();
+ for (Schema schema : schemas) {
+ NestedField retained = schema.findField(field.fieldId());
+ if (retained == null) {
+ continue;
+ }
+ optional |= retained.isOptional();
+ if (type.isPrimitiveType() && retained.type().isPrimitiveType()
+ && isWideningPromotion(type.asPrimitiveType(),
retained.type().asPrimitiveType())) {
+ type = retained.type();
+ }
+ }
+ if (type.isStructType()) {
+ type =
Types.StructType.of(withRetainedRelaxations(type.asStructType().fields(),
schemas));
+ }
+ relaxed.add(type.equals(field.type()) && optional ==
field.isOptional()
+ ? field :
Types.NestedField.from(field).ofType(type).isOptional(optional)
+
.withInitialDefault(promotedDefault(field.initialDefaultLiteral(),
field.type(), type))
+
.withWriteDefault(promotedDefault(field.writeDefaultLiteral(), field.type(),
type))
+ .build());
+ }
+ return relaxed;
+ }
+
+ /**
+ * {@link TypeUtil#isPromotionAllowed} plus the v3 promotion of DATE to an
unzoned TIMESTAMP or TIMESTAMP_NS,
+ * which the SDK's schema update does not implement yet but the format
allows.
+ */
+ private static boolean isWideningPromotion(Type.PrimitiveType from,
Type.PrimitiveType to) {
+ if (TypeUtil.isPromotionAllowed(from, to)) {
+ return true;
+ }
+ if (from.typeId() != Type.TypeID.DATE) {
Review Comment:
[P1] Follow UNKNOWN promotions when recovering a dropped equality key
V3 also permits `UNKNOWN` to become any concrete type ([promotion
table](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/format/spec.md#schema-evolution)).
The pinned SDK helper does not recognize that edge, and this non-DATE return
leaves a recovered UNKNOWN unchanged.
Use an unpartitioned v3 table with optional `k UNKNOWN` and commit S1 with k
omitted. Publish schema-only UNKNOWN→DATE, prepare a DATE append containing
`k=2024-01-02`, publish schema-only DATE→TIMESTAMP_NS, prepare a matching
midnight nanos equality delete, then drop k and commit the pending append
followed by the delete. Publish the schemas through `TableMetadata.Builder`, as
the temporal tests do, and omit all column metrics so pruning retains the
delete. Both pending snapshots record the dropped schema, so lineage recovery
finds only S1's UNKNOWN.
Retained DATE/NANO definitions never replace UNKNOWN. `IcebergSchemaUtils`
encodes it as STRING; V2 restores that carrier type and casts both keys to
strings. The DATE becomes `2024-01-02`, while the matching tombstone becomes
`2024-01-02 00:00:00.000000000`, so the deleted row remains visible. The
previous reverse metadata lookup selected the retained nanos definition and
matched correctly.
Recognize UNKNOWN→concrete promotions and cover this pending-writer history.
This differs from the existing DATE-origin thread: the new helper never reaches
its DATE handling when the lineage definition is UNKNOWN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
