SabyaXSachi opened a new pull request, #55662:
URL: https://github.com/apache/arrow/pull/55662

   # GH-55650: [C++][ORC] Preserve offsets when normalizing structs
   
   ### Summary
   
   The ORC writer serializes a sliced `struct` column with struct-level nulls 
incorrectly: it writes the struct's validity and the child values starting at 
physical row 0 instead of at the array's offset. No error is raised — the file 
itself is wrong, and the struct column ends up misaligned with the other 
columns in the same file. This affects everyday operations such as 
`Table.slice()` / `table[a:b]`, or writing chunks produced by 
`to_batches(max_chunksize=...)`. The same happens when the struct has offset 0 
but a child array has its own independent offset.
   
   Example (pyarrow): writing `t.slice(1, 4)` of `[{a:0}, None, {a:2}, {a:3}, 
None, {a:5}]` produces `[{'a': 0}, None, {'a': 2}, {'a': 3}]` instead of 
`[None, {'a': 2}, {'a': 3}, None]`, while the `id` column is written correctly. 
The Apache ORC C++ reader returns the same wrong values, confirming the file is 
corrupt, not just the Arrow reader.
   
   ### Root cause
   
   `NormalizeArray()` in `cpp/src/arrow/adapters/orc/util.cc` (STRUCT branch) 
runs when `null_count() != 0` and rebuilds each child to AND in the struct's 
validity. In doing so it drops every offset:
   
   1. `StructArray::field(i)` returns the child already re-sliced to the 
struct's range (child offset = child offset + struct offset), but
   2. `BitmapAnd()` reads both validity buffers from bit 0 instead of from the 
parent's and child's offsets,
   3. `ArrayData::Make(child->type(), child->length(), ...)` rebuilds the child 
without passing `child->offset()`, so it defaults to 0 over the original, 
unsliced buffers, and
   4. `std::make_shared<StructArray>(...)` drops the struct's own offset while 
reusing the unsliced validity buffer.
   
   Additionally, when a child has no validity bitmap, the parent's raw, 
unoffset bitmap buffer was assigned directly as the child's validity buffer, 
which only aligns by coincidence.
   
   GH-15212 / PR #15213 fixed the same class of bug for the 
LIST/LARGE_LIST/FIXED_SIZE_LIST/MAP branches by passing offsets through; the 
STRUCT branch was left as it was. The existing `writeSlicedBatch` regression 
test does not reach this branch because it generates its table with 
`probability=0` (no nulls), so `NormalizeArray` takes the `null_count() == 0` 
early return.
   
   ### Fix
   
   Use the existing `StructArray::GetFlattenedField()` abstraction — the same 
one the SPARSE_UNION branch of this function already uses. It pushes the parent 
slice down to each child (child offset = child offset + parent offset) and 
merges parent and child validity at the correct bit positions, including the 
case where the child has no validity bitmap. Each flattened child is normalized 
recursively. The struct's own validity bitmap is then rebased to offset 0 with 
`AllocateEmptyBitmap()` + `CopyBitmap()`, and the struct is rebuilt at offset 0 
with its correct length and null count, matching the now offset-free children. 
The `null_count() == 0` early return is unchanged.
   
   ### Tests
   
   Deterministic regression tests added to 
`cpp/src/arrow/adapters/orc/adapter_test.cc`:
   
   - `writeSlicedBatchWithStructNulls`: a struct column with struct-level 
nulls, sliced with a non-zero offset (`table->Slice(1, 4)`), round-tripped 
through the ORC writer/reader.
   - `writeStructWithOffsetChild`: struct at offset 0 with a child that has its 
own offset (logical values `[13, 14, 15]` over physical `[10, 11, 12, 13, 14, 
15]`) and a struct-level null at row 1, round-tripped.
   - `writeSlicedStructWithOffsetChild`: combines a non-zero struct offset with 
an independently offset child.
   
   Results against the unpatched implementation (built locally at the same 
revision):
   
   ```
   [  FAILED  ] TestORCWriterTrivialNoConversion.writeSlicedBatchWithStructNulls
   [  FAILED  ] TestORCWriterTrivialNoConversion.writeStructWithOffsetChild
   [  FAILED  ] 
TestORCWriterTrivialNoConversion.writeSlicedStructWithOffsetChild
   [  PASSED  ] 0 tests.
   ```
   
   The failure diffs reproduce the corruption reported in the issue exactly 
(e.g. child values `[10, null, 12]` written instead of `[13, 14, 15]`, validity 
shifted from `[false, true, true, false]` to `[true, false, true, true]`). 
After this patch, all three pass:
   
   ```
   [       OK ] TestORCWriterTrivialNoConversion.writeSlicedBatchWithStructNulls
   [       OK ] TestORCWriterTrivialNoConversion.writeStructWithOffsetChild
   [       OK ] 
TestORCWriterTrivialNoConversion.writeSlicedStructWithOffsetChild
   [  PASSED  ] 3 tests.
   ```
   
   ### Validation
   
   - Build: `cmake -G Ninja` with `-DARROW_ORC=ON -DARROW_BUILD_TESTS=ON 
-DARROW_COMPUTE=ON -DARROW_JSON=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED` (Release, 
MSVC x64), target `arrow-orc-adapter-test`.
   - Full ORC adapter suite: `ctest -R arrow-orc-adapter-test` — **100% tests 
passed, 0 failed (44/44)** with this patch.
   - Without the patch, the same suite passes everything except the three new 
tests (the remaining failures were a local environment issue: the ORC writer 
needs an IANA time zone database on Windows, fixed locally by pointing `TZDIR` 
at an existing zoneinfo tree; those tz-dependent tests fail identically with 
and without this patch).
   - `git diff --check`: clean. No unrelated tests were modified.
   
   ### Scope
   
   This PR addresses GH-55650 only. GH-55618 (a struct-normalization bug on the 
`null_count() == 0` early-return path of the same function) is related but 
distinct and is intentionally not addressed here.
   
   ### AI disclosure
   
   This contribution was developed with the assistance of an AI coding agent. 
The complete diff was reviewed by the author, who verified the root-cause 
analysis against the checked-out sources, and the build and regression tests 
were run locally with the results reported above.
   
   Closes #55650.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to