sunshone opened a new issue, #13615:
URL: https://github.com/apache/gluten/issues/13615

   ### Description
   
   **Description**:
   
   Proposal: support the Flink Rank operator family on GlutenFlink (Flink on 
the velox stateful backend), so rank-family queries translate to the velox 
stateful backend and produce output exactly matching native Flink.
   
   Supported shapes (round 1 — grouped/global TopN and Top1):
   
   - Grouped TopN (`ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) WHERE rn 
<= N`), with and without the rank number in the output.
   - Global TopN (no `PARTITION BY`) via a sentinel key spec.
   - Grouped Top1 (`rn <= 1`) with the Flink FastTop1-style changelog (-U/+U 
instead of -D/+I).
   - Global Top1.
   
   Supported shapes (round 2 — spec gap closure):
   
   - FirstN fast path: single proctime ASC sort key routes to a per-key counter 
with append-only, +I-only emission; `ORDER BY proctime_col LIMIT N` routes 
through the same branch.
   - Offset rank ranges: `WHERE rn BETWEEN K AND N`, `rn = K`, `LIMIT N OFFSET 
K` — true rankStart/rankEnd pass through, emissions gated by isInRankRange, 
with-row-number algorithm forced when the range has an offset.
   - Bare `ORDER BY ... LIMIT [OFFSET K]` (no window) through the 
StreamExecSortLimit subclass binding.
   - Sort-key types: 13 logical families (11 scalar velox kinds + DECIMAL p<=38 
+ DATE); TIMESTAMP_LTZ normalizes into TIMESTAMP.
   - Deduplicate boundary made explicit: rank dedup shapes fail loud at 
translation time with a clear message (native dedup tracked separately).
   - Multi-parallelism checkpoint restore: the native keyed state backend owns 
exactly the subtask's key-group range and HASH exchanges route through the 
native key-group partitioner — savepoint resume shards correctly at any 
parallelism.
   
   Semantics preserved: AppendFast insert-only processing with per-key TopN 
eviction chains; changelog kinds (+I/-U/+U/-D) matching Flink's 
`AppendOnlyTopNFunction`/`FastTop1Function`/`AppendOnlyFirstNFunction`, 
including `generateUpdateBefore` gating in both settings; ties resolved in 
arrival order; Flink null-ordering rules for sort keys; HEAP typed keyed-state 
checkpoint / savepoint resume equivalence at parallelism 1 and 4 (restored 
output equals a continuous run).
   
   Known limitations (all fail loud at translation time, not silent 
misbehavior):
   
   - UpdateFast/Retract update strategies (update-stream family).
   - Rank ranges without a constant end (streaming RANK()/DENSE_RANK(), 
VariableRankRange).
   - TIME sort keys (rejected at the type converter, which runs before the rank 
guard).
   - A non-default `pipeline.max-parallelism` (only the default 128 is wired 
through the partition spec and backend).
   - Deduplicate native implementation (rank dedup shapes guard-rejected).
   - ROCKSDB state backends; WindowRank; TTL/LRU state pruning; null partition 
keys; e2e-level cross-parallelism rescale (covered at unit-test level only).
   
   **Gluten version**: main branch (workspace gluten-flink built as 
1.9.0-SNAPSHOT)
   
   ### Gluten version
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to