cshuo commented on issue #19025:
URL: https://github.com/apache/hudi/issues/19025#issuecomment-5403954502
## Proposed Design
### Target Architecture
```text
HoodieMetadataTableServicesTool [NEW]
└─ run()
│
▼
HoodieTableMetadataWriter [EXISTING]
├─ scheduleTableServices(request) [NEW API]
└─ executeTableServices(request) [NEW API]
│
▼
Existing MDT write client APIs [EXISTING]
├─ Scheduling
│ ├─ scheduleCompactionAtInstant()
│ └─ scheduleLogCompactionAtInstant()
│
└─ Execution
├─ compact(instant, true)
├─ logCompact(instant, true)
├─ runAnyPendingCompactions()
├─ runAnyPendingLogCompactions()
├─ clean()
└─ archive()
```
The proposal is to add a standalone Spark-submit tool named
`HoodieMetadataTableServicesTool` under `hudi-utilities`.
The tool takes the **data-table base path and write configs** as input and
runs the same MDT maintenance flow currently performed inline by
`HoodieBackedTableMetadataWriter#performTableServices`, without requiring an
empty data-table commit.
### Shared MDT Table-Service Flow
Instead of duplicating `performTableServices` logic in the tool, refactor it
into reusable, phase-oriented APIs on `HoodieTableMetadataWriter`, for example:
```java
MetadataTableServicePlan scheduleTableServices(
MetadataTableServiceRequest request);
MetadataTableServiceResult executeTableServices(
MetadataTableServiceRequest request);
```
The existing `performTableServices()` remains the inline-writer entry point
and delegates to the same implementation. The standalone tool invokes it using
an external-tool request.
A request should describe:
```text
mode: SCHEDULE | EXECUTE | SCHEDULE_AND_EXECUTE
services: compaction, logcompaction, clean, archive
optional instant to execute
```
The default tool behavior should be:
```text
services = all
mode = SCHEDULE_AND_EXECUTE
```
This makes it functionally equivalent to the current inline
`performTableServices()` flow.
### Locking Model
The tool uses two MDT writer profiles for different phases:
```text
Scheduling writer
- MDT concurrency mode: SINGLE_WRITER
- Used only while the tool holds the data-table lock
- Does not acquire an additional MDT lock
Execution writer
- MDT concurrency mode: OPTIMISTIC_CONCURRENCY_CONTROL
- Used without an outer data-table lock
- Uses the shared data-table lock only for short state transitions.
```
The table-service flow is split into the following lock-aware phases:
```text
1. Execute existing pending compaction/log-compaction plans
- Use the OCC MDT writer
- Do not hold an outer data-table lock
- Use the concurrency support added by #18295
- Acquire short locks only around claim/heartbeat and completion
2. Validate, clean, and schedule
- Acquire the data-table lock
- Create or use a SINGLE_WRITER MDT writer
- Reload the DT and MDT timelines
- Run the existing scheduling validation
- Compute the safe MDT compaction instant
- Schedule compaction/log-compaction plans
- Run clean under the current locking model
- Close the scheduling writer and release the data-table lock
3. Execute newly scheduled plans
- Create or use the OCC MDT writer
- Do not hold an outer data-table lock
- Use existing compact(instant, true) /
logCompact(instant, true) execution behavior
4. Archive
- Reacquire the data-table lock
- Create or use a SINGLE_WRITER MDT writer
- Reload the MDT timeline and archive
- Release the data-table lock
```
This avoids the reentrant lock issue while preserving atomicity between
DT/MDT timeline validation and table-service plan publication, and preserves
the ordering of the current flow as well:
```text
pending services → clean → compaction/log-compaction → archive
```
while avoiding holding the data-table lock during expensive compaction
execution.
The existing `SparkRDDMetadataWriteClient` compaction and completion APIs
should not require changes. #18295 already provides the required heartbeat,
pending-instant validation, short locking, and finalization behavior.
### Delegating Inline Table Services
Keep the existing config for execution delegation:
```properties
hoodie.metadata.table.service.manager.actions=compaction,logcompaction,clean,archive
```
Add one config for scheduling delegation:
```properties
hoodie.metadata.table.service.manager.schedule.actions=compaction,logcompaction
```
The existing config remains backward compatible, while the new config allows
ingestion writers to skip inline scheduling.
Archive has no scheduling phase. Since clean currently combines scheduling
and execution, delegating `clean` skips the entire inline clean operation.
The external tool is itself the table-service manager, so it should follow
the request mode rather than re-applying the ingestion-side delegation checks.
### CLI Example
```bash
spark-submit \
--class org.apache.hudi.utilities.HoodieMetadataTableServicesTool \
hudi-utilities-bundle.jar \
--base-path s3://bucket/data-table \
--services all \
--mode schedule-and-execute \
--props table-service.properties
```
### Implementation Plan
1. Add `scheduleTableServices()` and `executeTableServices()` APIs to
`HoodieTableMetadataWriter`.
2. Refactor `HoodieBackedTableMetadataWriter#performTableServices()` based
on the new APIs and a shared internal implementation, without changing existing
inline behavior.
3. Add `HoodieMetadataTableServicesTool` under `hudi-utilities` with
Spark-submit support, using a SINGLE_WRITER MDT writer under the data-table
lock for scheduling and an OCC MDT writer for compaction execution.
4. Add an ingestion-side scheduling delegation config. Continue using the
existing table-service manager actions config for execution delegation, while
allowing the external tool to bypass ingestion-side delegation checks.
5. Preserve the version-specific scheduling semantics implemented by
`HoodieBackedTableMetadataWriter` and
`HoodieBackedTableMetadataWriterTableVersionSix`, including instant generation
and compaction/log-compaction fallback behavior.
6. Optimize clean execution to run outside the long-held data-table lock in
a follow-up if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
