AndreaBozzo commented on PR #10579: URL: https://github.com/apache/arrow-rs/pull/10579#issuecomment-5215984188
One decision here is worth flagging separately, since the issue did not cover it. Skipped rows go through `RecordDecoder::decode` as well, so a header row shorter than the schema gets padded like any other short row. I chose not to count those: `Decoder::decode` discards skipped rows via `clear()`, and `clear()` now resets the counter along with the buffered rows it refers to, so the total only covers rows that reached a batch. That seemed right for the reporting use case, but the opposite reading is defensible, that the count should be a raw tally of every row the decoder padded. If you prefer that, it is a one line revert of the reset in `clear()` plus a doc change, and `test_truncated_row_count_excludes_skipped_rows` inverts to assert 1. Happy to switch it either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
