zeroshade opened a new pull request, #1608:
URL: https://github.com/apache/arrow-go/pull/1608

   ### Rationale for this change
   
   Builds on #1601; only the last commit is new here.
   
   Repeated columns still read the wrong rows in three cases, all of which are 
also on main:
   
   - When V1 pages aren't aligned to rows, which is the writer's default 
without a page index, `Skip` stops at the end of a page once it has counted 
enough rows. The rest of the last row continues on the next page and is never 
skipped.
   - Without an offset index, `SeekToRow` counts V1 `num_values` as rows. For a 
repeated column that is the number of levels, so the seek lands on the wrong 
page and skips the wrong number of rows.
   - `ReadBatch` with nil repetition levels doesn't advance the repetition 
decoder, so later reads and skips on that page get the wrong levels.
   
   ### What changes are included in this PR?
   
   - `skipRows` keeps skipping after a page ends inside the last row it 
counted, until the next row starts.
   - Without an offset index, `SeekToPageWithRow` stops at the first V1 data 
page of a repeated column, and the column reader counts rows from there. 
Earlier V2 pages are still skipped by their row counts. Encrypted page readers 
now record the column's maximum repetition level as well.
   - `ReadBatch` without repetition levels caches the rest of the page's 
repetition levels, using the same cache that `skipRows` uses.
   
   ### Are these changes tested?
   
   Yes.
   
   ### Are there any user-facing changes?
   
   Yes. `Skip`, `SeekToRow` and `ReadBatch` without repetition levels now 
return the right rows for repeated columns with V1 pages. There is no API 
change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to