danny0405 commented on issue #19823:
URL: https://github.com/apache/hudi/issues/19823#issuecomment-5535406799
There are two reasonable ways to represent nested ordering fields in the
native delete-log schema. Both should resolve the configured field against the
table schema with `getNestedField` at write time; they differ in the physical
schema written to the delete file.
### Option 1: dedicated flat ordering columns
Compose an internal delete schema whose ordering columns are independent of
the source paths:
```text
record hudi_delete_log_record {
string _hoodie_record_key;
<nullable leaf type> _hoodie_ordering_0;
<nullable leaf type> _hoodie_ordering_1;
...
}
```
For example, ordering fields `event.metadata.timestamp,source.sequence`
become `_hoodie_ordering_0,_hoodie_ordering_1`. The source paths are needed
only to extract the values and resolve their leaf types. They do not need to be
persisted: the JSON/file schema preserves field order, so the reader can
reconstruct a scalar ordering value or `ArrayComparable` from the physical
ordering columns in schema order.
Pros:
* The delete file remains small and structurally simple.
* `createDeleteLogFieldValues` remains a flat top-level `Object[]`.
* There is no dotted-name escaping problem or collision such as `a.b_c`,
`a_b.c`, and `a_b_c` all becoming the same name.
* The file representation is decoupled from the table's nested structure;
only the ordering values and their physical types are retained.
* Multiple fields and fields sharing the same parent require no special
schema merging.
Cons:
* This introduces a native-delete-specific physical naming contract.
* Readers must use the physical delete-schema field names/order rather than
calling `getOrderingValue` with the original table ordering paths.
* The files are less immediately readable during manual inspection.
* If applied to existing top-level ordering fields too, it changes their
current physical schema. We could alternatively preserve the existing fast
path/schema for top-level-only configurations and use dedicated names only when
nesting is present.
### Option 2: projected nested schema (similar to Iceberg equality deletes)
Keep the ancestors of a nested field, but include only selected ordering
leaves:
```text
record hudi_delete_log_record {
string _hoodie_record_key;
record event {
record metadata {
nullable long timestamp;
}
}
}
```
Unselected siblings are omitted rather than written as null. If multiple
ordering fields share an ancestor, their projections must be merged into one
struct. This is similar to Iceberg equality-delete files, which may contain a
projection of the table schema and retain the nested structure (Iceberg
additionally uses table field IDs to identify equality columns):
https://iceberg.apache.org/spec/#equality-delete-files
Pros:
* The delete schema remains a conventional projection of the table schema.
* Existing dotted-path reads can continue to use `event.metadata.timestamp`;
no logical-to-physical field-name translation is necessary.
* The file is easier to inspect and preserves the meaning of each value.
* Parquet's storage penalty for a struct containing one leaf is small:
values are stored in the leaf column, with nesting represented mainly by schema
metadata and definition levels.
* We can leave the current flat implementation unchanged for the common
top-level-only case and invoke nested construction only when a nested ordering
field exists.
Cons:
* The writer must recursively build engine-native nested records
(`GenericData.Record`, `GenericInternalRow`, `GenericRowData`, etc.). The outer
return value can remain `Object[]`, but a top-level element representing a
struct must itself be an engine-native record.
* Projected schemas for ordering fields with shared ancestors must be merged
and deduplicated correctly.
* Cross-engine testing is more involved.
* Array/map traversal should probably be rejected initially unless ordering
semantics for such paths are explicitly supported.
Given that nested ordering fields are expected to be uncommon, Option 2 is
viable without affecting the common path: precompute whether any ordering path
is nested, keep the existing schema/value construction byte-for-byte for
top-level fields, and use the recursive path only for nested configurations.
Option 1 is simpler internally and avoids nested engine-record construction,
but requires a new physical naming/reader convention.
Separately, as noted above, `HoodieFlinkRecord.doGetOrderingValue` currently
performs top-level lookup only. That extraction issue must be fixed
independently; changing the native delete-file schema alone will not make
nested ordering work for Flink.
Whichever option is chosen, useful regression coverage would include:
* one nested ordering field;
* mixed top-level and nested ordering fields;
* two nested fields sharing an ancestor;
* a native delete-file write/read round trip;
* a regression assertion that top-level-only ordering retains its current
path and performance;
* the equivalent table-v8 inline-log scenario to confirm consistent merge
results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
