nsivabalan commented on issue #18135:
URL: https://github.com/apache/hudi/issues/18135#issuecomment-6422588922
Thanks @shangxinli @vamshipasunuru1. Sharing a proposal for the requirements
above, building on @vinothchandar's suggestion to drive this through `FILES`.
We have a working prototype locally (Spark, COW, with and without the metadata
table).
### At a glance
```
Layout
/data/hive_table/datestr=2022-01-01/part-0.parquet <- cold,
untouched, never copied
/data/hudi_table/
.hoodie/hoodie.properties <- table marked
as zero copy bootstrapped (see open question)
.hoodie/.aux/.zero_copy_bootstrap/<instant>.json <- registry:
{datestr=2022-01-01 -> /data/hive_table}
.hoodie/metadata/ FILES: datestr=2022-01-01 ->
part-0.parquet_<instant>_hudiext
datestr=2026-10-01/... <- hot, native
Hudi
(nothing for datestr=2022-01-01)
Registration (admin operation)
list datestr=2022-01-01 at /data/hive_table (names + sizes only, no
reads)
-> write registry snapshot <instant>.json
-> complete replacecommit <instant> with the listed files as write stats
-> MDT FILES gets the partition and its files
Writes
insert / upsert / delete into datestr=2022-01-01 -> rejected
any write into native partitions -> as today
cleaner on a replaced registered file group -> removes the entry,
never deletes the file
delete_partition datestr=2022-01-01 -> unregisters it (new
registry snapshot), deletes nothing
Reads (WHERE datestr = '2022-01-01')
with MDT: FILES -> part-0.parquet_<instant>_hudiext
registry says datestr=2022-01-01 lives at /data/hive_table
-> read /data/hive_table/datestr=2022-01-01/part-0.parquet
(meta columns null)
without MDT: partitions = listing under table ∪ registry
-> list /data/hive_table/datestr=2022-01-01 at query time,
only if pruning keeps it
```
### Details
**Registry.** One immutable snapshot per admin instant under
`.hoodie/.aux/.zero_copy_bootstrap/`, each holding all registered partitions:
```
{"version":1,"partitions":{"datestr=2022-01-01":{"basePath":"/data/hive_table","instantTime":"<instant>"}}}
```
A reader uses the newest snapshot whose instant is completed, so a failed
registration is never visible; rollback of the instant deletes its snapshot. No
change to `.hoodie_partition_metadata`, no change to the metadata table schema,
nothing written under the table base path for a registered partition.
**Registration.** Never opens a cold file (R1). File names use the existing
externally-created naming, `<file>_<instant>_hudiext`, so file groups and
visibility come from existing code paths. With the MDT on, the commit fills
`FILES` and the partition list, so queries never list the external path. A
partition is either fully registered or fully native (R3).
**Reads.**
- With the MDT, the only change is where the absolute file path is built:
under the registry's base path instead of the table base path.
- Without the MDT, registered partitions are added to the partition listing
and their files are listed from the old path at query time.
- Meta columns come back null for registered files, as already proposed here.
- These tables skip the `storage.exists(<table>/<partition prefix>)`
shortcut in Spark partition pruning, and partition-stats pruning keeps
partitions that have no stats.
**Writes.** The ingestion check runs before records are routed to file
groups, so an insert is never sized into a registered file as an update, which
would read it. Details for `insert_overwrite` into registered partitions (R12)
are being worked on and will follow.
**Follow-ups:** clustering/compaction skipping registered partitions, MOR,
Presto/Trino readers (per @vamshipasunuru1), and a table version gate for older
readers.
### Open questions to get consensus on
**1. Table property.** The prototype marks the table with a new property,
`hoodie.bootstrap.zero.copy.enable`. We did not reuse
`hoodie.bootstrap.base.path`: today it means a METADATA_ONLY/FULL_RECORD
bootstrapped table, several read and write paths branch on it, and a registered
partition carries its own base path. The alternatives:
- (a) A new property, as in the prototype. Simple, but one more persisted
property to support forever.
- (b) Reuse `hoodie.bootstrap.base.path` and add a mode/granularity property
to tell the two bootstrap kinds apart. Needs care so existing bootstrap code
paths don't misread these tables.
A table property is a persisted choice that's hard to undo, so we'd like
input from other PMC members and bootstrap experts before settling on one. cc
@vinothchandar @bhasudha @prashantwason
**2. Which partition categories are needed in one table?** This proposal
supports two: registered (external, read in place) and native Hudi partitions.
A table with registered partitions rejects METADATA_ONLY (skeleton-based)
bootstrap. @shangxinli @vamshipasunuru1, is that enough for your use case, or
do you also need METADATA_ONLY-bootstrapped partitions alongside the other two
in the same table? Supporting all three would affect the table property
question above, as well as the read and write paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
