ggjh-159 opened a new issue, #12459: URL: https://github.com/apache/gluten/issues/12459
### Description ## Background Follows up on the design note in [apache/gluten#12164](https://github.com/apache/gluten/discussions/12164) (Gluten Flink Pending tasks), where retraction semantics is called out as a pending gap: > retraction semantics — this feature is required for group aggregate and join, which has not been implemented, maybe we should introduce the `rowkind` just as defined in `flink`, combine it with row vector. Flink `RowData` carries a `RowKind` enum (`+I` / `-U` / `+U` / `-D`) that distinguishes changelog records — insert / update-before / update-after / delete. Gluten-Flink hands RowData to the velox C++ engine via the RowData → RowVector conversion path, but `RowKind` is dropped during that conversion today. As a result, the velox side sees every row as `+I` regardless of the original changelog marker. ## Problem - Sink operators (`PrintSinkFactory`, `FileSystemSinkFactory`, etc.) can't emit correct `-U` / `+U` / `-D` prefixes for upsert / delete streams — outputs are uniformly `+I[...]`. - Retract aggregations, temporal joins, and other row-kind-aware operators lose the row-kind signal once data crosses into velox. - Nexmark / E2E tests over upsert streams produce incorrect changelog output. ## Proposal Encode `RowKind` as a synthetic column that flows through the entire plan and is interpreted at three layers: the Java ↔ C++ boundary, the velox stateful operators that must act on changelog semantics, and the sink layer that emits the changelog prefix. - **Type**: `TINYINT` (Flink's `RowKind` enum ordinal: 0=INSERT, 1=UPDATE_BEFORE, 2=UPDATE_AFTER, 3=DELETE). - **Column name**: reserved placeholder, invisible to user SQL — exact convention TBD (candidates: `__row_kind$`, `$row_kind$`). - **Position**: trailing column appended by the conversion layer. The Row-kind-aware operators in velox (retract aggregation, temporal/regular join, top-N, deduplication, etc.) must read it to apply insert / update-before / update-after / delete semantics to their state. Stateful operators that ignore the column would either treat `-U` / `-D` as `+I` (incorrect accumulation) or drop them silently — both wrong. ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
