bharos opened a new issue, #11699:
URL: https://github.com/apache/gravitino/issues/11699
### Describe the feature
Add end-to-end support for a **sort** maintenance strategy for Iceberg
tables in the Table Maintenance Service (Optimizer), so a table can be
automatically rewritten into a sorted (or z-ordered) data file layout based on
a policy — analogous to the existing `system_iceberg_compaction` strategy.
The execution layer already exists: the built-in
`builtin-iceberg-rewrite-data-files` job (`IcebergRewriteDataFilesJob`) already
supports `--strategy sort` and `--sort-order`. What is missing is the **policy
+ strategy-recommendation layer** that decides *when* a table should be sorted
and emits the corresponding job.
### Motivation
Sorting (and z-ordering) data files significantly improves query performance
via better data skipping and file pruning, especially for high-cardinality
filter columns. Today the optimizer only has a compaction (`binpack`) strategy
(`CompactionStrategyHandler` / `system_iceberg_compaction`); there is no way to
express "keep this table sorted by columns X, Y" as a managed policy and have
the optimizer schedule it.
This complements the in-flight expire-snapshots (#11194),
remove-orphan-files (#11195), and rewrite-manifests (#11196) work under
umbrella #8864, reusing the same strategy → job design pattern established in
the expire-snapshots design doc (#11206).
### Describe the solution
Following the existing compaction flow and the design pattern in #11206:
1. **Policy content** — add `IcebergSortContent` (similar to
`IcebergDataCompactionContent`) with: sort order (list of `{column, direction,
nullOrdering}`), sort `strategy` (`sort` | `zorder`), optional `where` filter,
and rewrite options (e.g. `target-file-size-bytes`, `min-input-files`).
2. **Policy type** — add `system_iceberg_sort` to `Policy.BuiltInType` and
the matching `PolicyContentDTO` / `PolicyDTO` JSON subtypes (mirroring
`system_iceberg_compaction`).
3. **Strategy handler** — add `SortStrategyHandler extends
BaseExpressionStrategyHandler` (mirroring `CompactionStrategyHandler`),
emitting a `JobExecutionContext` that invokes
`builtin-iceberg-rewrite-data-files` with `strategy=sort` and the configured
`sort-order`. Register it in `GravitinoStrategyProvider`.
4. **Reuse** the existing `IcebergRewriteDataFilesJob` — no new job class is
needed.
5. Unit tests for the policy content (serde), the strategy handler
(recommendation logic), and the job-options mapping, consistent with the
existing compaction / expire-snapshots test coverage.
### Additional context
- Umbrella: #8864 (built-in jobs for table maintenance)
- Sibling strategies: #11194 (expire-snapshots), #11195
(remove-orphan-files), #11196 (rewrite-manifests)
- Design pattern reference: #11206 (expire-snapshots end-to-end design doc)
- Trino-side `rewrite_data_files` (incl. sort) procedure delegation already
merged in #10500
I'd like to work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
