roryqi opened a new issue, #11672:
URL: https://github.com/apache/gravitino/issues/11672
### What would you like to be improved?
The `maintenance/optimizer` recommender's trigger-expr / score-expr
evaluation has two limitations:
1. **Limited evaluation context** — Trigger expressions can currently only
reference partition/table statistics. They cannot reference table metadata
(column count, partition count, sort-order count, table properties), which
restricts the kinds of rules users can write.
2. **Inefficient partitioned-table evaluation** — Every partition is
re-evaluated even when a table-level expression already decides the outcome,
and the table-level context is rebuilt for every partition. This is costly for
large partitioned tables. Additionally, `QLExpressionEvaluator` calls
`Pattern.compile` on every evaluation.
### How should we improve?
1. Extend the trigger-expr context to expose `column_count`,
`partition_count`, `sort_order_count`, and table properties (numeric values
parsed to `long`, others kept as `string`), alongside existing partition/table
statistics. Both partitioned and non-partitioned tables evaluate against
partition statistics (when present), table statistics, and table metadata. The
trigger-expr string representation stays unchanged.
2. Speed up partitioned-table evaluation (port of Pinterest
gravitino-pinterest#249):
- Short-circuit: evaluate with table-level context only; if it resolves
without referencing partition variables, skip the per-partition loop.
- Precompute the table-level context once per `initialize()`.
- Cache compiled hyphen-to-underscore regex patterns in
`QLExpressionEvaluator`.
- Add `ExpressionEvaluator#tryToEvaluateBool` returning
`Optional<Boolean>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
