hcrosse commented on code in PR #1585:
URL: https://github.com/apache/arrow-go/pull/1585#discussion_r4591784024
##########
parquet/file/column_reader.go:
##########
@@ -670,13 +671,13 @@ func (c *columnChunkReader) skipRows(nrows int64) error {
ndefs, nvals, err := c.determineNumToRead(toSkip, nil,
nil)
if err != nil {
c.err = err
- return err
+ return nrows - toSkip, err
}
skipped, err := c.curDecoder.Discard(int(nvals))
if err != nil {
c.err = err
- return err
+ return nrows - toSkip, err
Review Comment:
The count includes only rows skipped before the step that failed. The reader
stops at the error, so a more exact count would not help a caller resume. The
`Skip` doc now says so, and `TestSkipStopsAtDecodeError` has a case with a
valid value before the bad one that checks its row is not counted.
##########
parquet/file/column_reader.go:
##########
@@ -729,14 +730,14 @@ func (c *columnChunkReader) skipRows(nrows int64) error {
_, err = c.curDecoder.Discard(int(valuesToSkip))
if err != nil {
c.err = err
- return err
+ return nrows - toSkip, err
Review Comment:
The count includes only rows skipped before the step that failed, so a row
whose values were partly discarded when the error occurred is not counted. The
reader stops at that error, and the `Skip` doc now says so.
##########
parquet/file/column_reader_types.gen.go.tmpl:
##########
@@ -30,9 +30,13 @@ type {{.Name}}ColumnChunkReader struct {
// Skip skips the next nvalues so that the next call to ReadBatch
// will start reading *after* the skipped values.
+//
+// It returns the number of rows skipped before it stopped, which is less than
+// nvalues when the column ends or an error occurs first. After an error on a
+// repeated column, the count can include a row that continues onto the page
+// that failed.
Review Comment:
I used your wording in the template and all eight generated readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]