nevzheng commented on PR #12257:
URL: https://github.com/apache/gravitino/pull/12257#issuecomment-5186596691
note: Hi @roryqi and @lasdf1234 — thanks for the suggestion. This is a
material alternative
architecture, not a small revision to the current proposal.
### Recommended V2 model
I recommend retaining the current V2 model. It is grounded in prior
research, API and metadata
experimentation, and a tested POC.
### Principles and design choices
- **Separation of concerns and clarity.** A deletion record captures
retention, audit, restore,
and purge eligibility. A cleanup job claims and executes physical purge
work. Reusing the
existing purge worker is valuable, but making its job row also serve as
the retained deletion
record couples two state machines with different lifetimes and
responsibilities.
- **Narrow initial scope with deliberate extensibility.** This V2 proposal
initially applies only
to Iceberg tables exposed through IRC. I chose a flexible, more separated
data model that can
extend soft deletion to other object types through later, separately
reviewed work. I recognize
that this goes beyond the narrowest IRC-only implementation; I believe the
product optionality
is valuable, and ask @markhoerth and @jerryshao to weigh in on that
trade-off.
- **Iterate before replacing.** I am confident that substantive concerns can
be evaluated and
addressed within the current model before we replace it with a different
metadata model.
### Initial critique of the retained-job model
I do not yet have a complete alternative design to evaluate, so this is an
initial architectural
objection rather than a claim that the retained-job approach cannot work. My
concern is that it
models a logical deletion as a cleanup job.
- A deletion is a durable entity-lifecycle fact: the object was deleted, by
whom, when it expires,
whether it can be restored, and whether it is eligible for purge.
- A cleanup job is an operational fact: work has been scheduled or claimed,
it is running or
retrying, and it eventually succeeds or fails.
- A table should be deleted because it has a deletion record in a deleted
state, not because a
cleanup job happens to exist. The cleanup job should reference an eligible
deletion when
physical purge begins; it should not define the deletion itself.
Combining those concepts forces user-facing retention and recovery semantics
into a worker state
machine, and makes audit, visibility, restore, and retry behavior depend on
the lifecycle of an
execution row. I prefer the clearer relationship: a durable deletion record
may later create or
link to a purge job.
### Alternative and decision
The retained-`iceberg_cleanup_job` model is a valid alternative, but it
needs a comparable
metadata/API proposal before it can replace the current design. I cannot
lead a parallel design.
If @roryqi or @lasdf1234 would like this option evaluated, please document
its implementation
scope, data model, lifecycle, IRC visibility behavior, support for
unprojected IRC-visible tables,
restore/purge behavior, and trade-offs.
### Next steps
- @markhoerth and @jerryshao: please weigh in on two linked decisions:
1. Should V2 use a soft-delete model that is initially delivered only for
Iceberg tables through
IRC, but can be extended to other supported object types through later,
separately reviewed
work?
2. Should durable deletion, recovery, retention, and audit metadata remain
a clean, separate
lifecycle model rather than being combined with purge-worker execution
state in
`iceberg_cleanup_job`?
I recommend yes to both.
- @roryqi and @lasdf1234: if the retained-job model should be considered,
please provide the
comparable alternative proposal described above.
- I will update the design and implementation plan to align with the
resulting decision.
### Design Review Feedback
note: @roryqi, for future design reviews, I recommend leading with the
specific requirement,
correctness risk, or trade-off that the current design does not address,
rather than immediately
asking the author to pivot to a different architecture. That lets us first
test whether the
existing proposal can resolve the concern, and only then decide whether a
material metadata-model
change is warranted.
This approach keeps review actionable, makes trade-offs legible, and lets us
move forward as a
team without prematurely discarding completed design work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
