srijithr03 commented on code in PR #10903:
URL: https://github.com/apache/arrow-rs/pull/10903#discussion_r3889020284
##########
arrow-csv/src/reader/records.rs:
##########
@@ -96,16 +110,47 @@ impl RecordDecoder {
return Ok((0, 0));
}
- // Reserve sufficient capacity in offsets
- self.offsets
- .resize(self.offsets_len + to_read * self.num_columns, 0);
-
// The current offset into `input`
let mut input_offset = 0;
// The number of rows decoded in this pass
let mut read = 0;
+ // Resume skipping extra fields if we were in the middle of it from a
previous chunk
+ if self.skipping_extra_fields {
+ let mut dummy_data = [0u8; 128];
+ let mut dummy_ends = [0usize; 1];
+ loop {
+ let (res, b_read, _, _) = self.delimiter.read_record(
+ &input[input_offset..],
+ &mut dummy_data,
+ &mut dummy_ends,
+ );
+ input_offset += b_read;
+ match res {
+ ReadRecordResult::Record => {
+ self.skipping_extra_fields = false;
+ read += 1;
+ self.current_field = 0;
+ self.line_number += 1;
+ self.num_rows += 1;
+ break;
+ }
+ ReadRecordResult::OutputFull |
ReadRecordResult::OutputEndsFull => {}
+ ReadRecordResult::End | ReadRecordResult::InputEmpty => {
+ return Ok((read, input_offset));
+ }
+ }
+ }
+ if read == to_read || input.len() == input_offset {
+ return Ok((read, input_offset));
Review Comment:
Thanks! Addressed this by extracting the duplicated skip/resume logic into
the private `skip_extra_fields()` helper, which is now used both when resuming
after a chunk boundary and when handling `ExtraFields::Ignore`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
