jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248544320
##########
arrow-avro/src/reader/mod.rs:
##########
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+ // The loop stops when the batch is full, a schema change is staged,
+ // or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
- if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
- // A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
- // a schema change is detected or there are insufficient bytes
to read the next prefix.
- // A schema change requires a new batch.
- total_consumed += prefix_bytes;
- break;
+ match self.handle_prefix(&data[total_consumed..], hash_type)? {
+ None => {
+ // No prefix: decode one row.
+ let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+ total_consumed += n;
+ self.remaining_capacity -= 1;
+ }
+ Some(0) => {
+ // Detected start of a prefix but need more bytes.
+ break;
+ }
Review Comment:
My intention was to use `Some(0)` to cover a scenario where magic bytes were
read, but there wasn't enough remaining bytes in the buffer for the
fingerprint. My thought was to then return so the caller could add more bytes.
I did just realize I'd need to change a mistake on L217 though for that.
After digging into the Java implementation's code however I noticed they
throw an error for this scenario:
https://github.com/apache/avro/blob/0606c33514e084cc0e131ab1f12a1a91e68e07c1/lang/java/avro/src/main/java/org/apache/avro/message/BinaryMessageDecoder.java#L150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
