felipepessoto commented on issue #12377: URL: https://github.com/apache/gluten/issues/12377#issuecomment-5334394528
Root cause found, and it is an upstream Velox bug in the scan rather than anything in Delta or in the DV bitmap aggregator. `SelectiveStructColumnReaderBase::next()` has a branch for structs with **no child readers** -- a scan projecting only columns that are not read from the file. There it clears `outputRows_` and repopulates it only when there is a deletion, then sizes the result vector to `numValues` while sizing the synthesized fields from `outputRows()`. Since `useOutputRows()` is `scanSpec_->hasFilter() || hasDeletion()`, a scan with a filter and no deletion gets the empty `outputRows_`, and `_metadata.row_index` comes back with **zero rows** while the RowVector containing it reports N. That empty child is then wrapped in a dictionary of size N (`BaseVector::wrapInDictionary` does not validate indexes against base size), so the aggregator reads past the end of a zero-length buffer. The "row indexes" we have been seeing -- `9223372036854775807`, `-1`, `0xe43315c000007f00`, `0xb858830000000000` -- were never row indexes at all, just whatever heap memory followed. That explains why the value differed on every occurrence. The Delta DV path hits all three conditions at once: it scans only `_metadata.row_index`, `file_path` and a partition key, and predicate pushdown puts a filter on the partition key. Consistent with that, only the `...DVsPredPushOn` suites fail; the same test on `...DVsPredPushOff` passes. Upstream: * Issue: https://github.com/facebookincubator/velox/issues/18535 * Fix: https://github.com/facebookincubator/velox/pull/18536 ## Evidence Two Gluten PRs, same base commit and same pinned Velox, differing only by whether the upstream fix is applied. Both enable Velox's `debug.validate_output_from_operators`, which catches the malformed vector at the scan instead of waiting for the garbage value to happen to be out of range: | PR | change | Delta suite | |---|---|---| | #12783 | validation enabled | 5+ of 8 shards failing, `Output validation failed for [operator: TableScan, plan node ID: 0]` | | #12808 | same, plus velox#18536 | **all 8 shards green** | ## Two further notes **Why bumping or reverting Velox never changed the CI outcome.** `SelectiveStructColumnReader.cpp` and `SelectiveColumnReader.h` are unchanged between `dft-2026_08_05` and `dft-2026_08_17`; the only commit touching them in that range is an unrelated DWRF flat-map fix. The defect was never in the code being bumped. **The abort is the rare tail, not the whole problem.** The row-index bounds check only rejects negatives and values above `kMaxRepresentableValue`. Pointer-shaped values around 1.4e14 are positive and comfortably inside Delta's valid range, so they pass silently. Instrumenting the malformed read rather than the value showed ~176 out-of-bounds reads in a single test-suite pass, nearly all accepted. Suite results were identical with and without the fix apart from the validation failures, so those reads were not visibly changing outcomes here -- consistent with the affected task producing no output rows -- but I would not assume the impact is limited to a flaky crash. ## Next Once velox#18536 merges it will arrive here through the regular weekly Velox bump. After that lands, the DV bitmap entries can be removed from `flaky-error-patterns.txt` and this issue closed. #12783 and #12808 exist only to produce the evidence above and are not intended to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
