malinjawi commented on PR #12215: URL: https://github.com/apache/gluten/pull/12215#issuecomment-5313597882
> @zhztheplayer I couldn't reproduce it locally — including the real `*MergeIntoExtendedSyntax*DVs*` suites from Delta v4.2.0 on Spark 4.1 (200 tests, incl. `...DVsPredPushOnSuite`). The native `_metadata.row_index` values always matched vanilla exactly. > > That said, I don't think `9223372036854775807` was ever written or read as a row ID. Delta's own `RoaringBitmapArray.add` enforces the identical bound (`MAX_REPRESENTABLE_VALUE = 9223372030412324864`), so vanilla Delta would abort the same way if that value ever reached the aggregator. And the sibling signature we quarantined for the same root cause is `Delta bitmap row index cannot be negative: -6254810385378525259` — arbitrary, not a sentinel. > > So it looks like garbage reaching the aggregator's input column, not a real row index. Since it only shows up under the full Delta suite, I opened #12783 to attach the offending column (encoding, nulls, value distribution) to the error, which should tell us whether it's a real row-index column with one corrupt slot or the wrong column entirely. Agree with @felipepessoto everything we found also says garbage in the row-index column, not a real row ID. One upstream data point: [velox#9943](https://github.com/facebookincubator/velox/issues/9943) documents the row-index vector getting only outputRows().size() slots written, with the tail serving recycled buffer contents as valid non-null BIGINTs same huge positive/negative value shapes as our two signatures. Not yet confirmed live in our pinned Velox, but the mechanism and the plan-shape-dependent intermittency line up. Should pair well with the diagnostics in [#12783](https://github.com/apache/incubator-gluten/issues/12783). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
