laserninja opened a new issue, #13335: URL: https://github.com/apache/gravitino/issues/13335
### Describe the feature Extend the Table Maintenance Service to use measured maintenance outcomes when selecting future actions, prioritizing expected workload benefit within a configured compute budget. ### Motivation The optimizer already scores maintenance candidates and evaluates before-and-after metrics. Connecting these capabilities through persistent decision history would help avoid repeatedly performing expensive maintenance with little measurable benefit and make recommendations explainable. ### Describe the solution - Persist each recommendation’s input statistics, strategy, estimated resource cost, execution identity, and measured outcome. - Implement an opt-in selector that uses historical outcomes to rank eligible actions within a defined resource budget. - Start with Iceberg compaction and one query engine, using comparable workload metrics and maintenance compute consumption. - Include cooldowns, concurrency limits, and explicit handling of missing or inconclusive measurements. - Provide explanations for selected and deferred actions, including estimated benefit, cost, and uncertainty. - Support shadow mode to compare recommendations with existing policies before enabling execution. Validate with repeatable workloads comparing resource consumption and workload performance against the existing selector. Account for caching and workload changes rather than treating every before-and-after improvement as causal. ### Additional context Complements [#11851](https://github.com/apache/gravitino/issues/11851), which proposes the Optimizer Service runtime and REST APIs. This feature focuses on decision quality and feedback, building on the existing recommender and monitor extension points. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
