sunshone opened a new issue, #13615: URL: https://github.com/apache/gluten/issues/13615
### Description **Description**: Proposal: support the Flink Rank operator family on GlutenFlink (Flink on the velox stateful backend), so rank-family queries translate to the velox stateful backend and produce output exactly matching native Flink. Supported shapes (round 1 — grouped/global TopN and Top1): - Grouped TopN (`ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) WHERE rn <= N`), with and without the rank number in the output. - Global TopN (no `PARTITION BY`) via a sentinel key spec. - Grouped Top1 (`rn <= 1`) with the Flink FastTop1-style changelog (-U/+U instead of -D/+I). - Global Top1. Supported shapes (round 2 — spec gap closure): - FirstN fast path: single proctime ASC sort key routes to a per-key counter with append-only, +I-only emission; `ORDER BY proctime_col LIMIT N` routes through the same branch. - Offset rank ranges: `WHERE rn BETWEEN K AND N`, `rn = K`, `LIMIT N OFFSET K` — true rankStart/rankEnd pass through, emissions gated by isInRankRange, with-row-number algorithm forced when the range has an offset. - Bare `ORDER BY ... LIMIT [OFFSET K]` (no window) through the StreamExecSortLimit subclass binding. - Sort-key types: 13 logical families (11 scalar velox kinds + DECIMAL p<=38 + DATE); TIMESTAMP_LTZ normalizes into TIMESTAMP. - Deduplicate boundary made explicit: rank dedup shapes fail loud at translation time with a clear message (native dedup tracked separately). - Multi-parallelism checkpoint restore: the native keyed state backend owns exactly the subtask's key-group range and HASH exchanges route through the native key-group partitioner — savepoint resume shards correctly at any parallelism. Semantics preserved: AppendFast insert-only processing with per-key TopN eviction chains; changelog kinds (+I/-U/+U/-D) matching Flink's `AppendOnlyTopNFunction`/`FastTop1Function`/`AppendOnlyFirstNFunction`, including `generateUpdateBefore` gating in both settings; ties resolved in arrival order; Flink null-ordering rules for sort keys; HEAP typed keyed-state checkpoint / savepoint resume equivalence at parallelism 1 and 4 (restored output equals a continuous run). Known limitations (all fail loud at translation time, not silent misbehavior): - UpdateFast/Retract update strategies (update-stream family). - Rank ranges without a constant end (streaming RANK()/DENSE_RANK(), VariableRankRange). - TIME sort keys (rejected at the type converter, which runs before the rank guard). - A non-default `pipeline.max-parallelism` (only the default 128 is wired through the partition spec and backend). - Deduplicate native implementation (rank dedup shapes guard-rejected). - ROCKSDB state backends; WindowRank; TTL/LRU state pruning; null partition keys; e2e-level cross-parallelism rescale (covered at unit-test level only). **Gluten version**: main branch (workspace gluten-flink built as 1.9.0-SNAPSHOT) ### Gluten version None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
