jecsand838 commented on PR #7834: URL: https://github.com/apache/arrow-rs/pull/7834#issuecomment-3059322972
@scovich > > I don't think the decoder should support implicitly inferring a schema to be clear. It should be the responsibility of the caller to identify when the writer schema has changed and to call a method that updates the decoder > > How would the caller do that, out of curiosity? A quick skim of the wire protocol didn't turn up anything obvious. Does the sender embed a new header+schema that the caller should notice and respond to as they see fit? The caller can do this by being signaled to read a new .avsc file, retrieving an updated schema definition from a schema registry, or attempting to decode an avro header from the sender (which I think happens) using the `Header` decoder in `arrow-avro/src/reader/header.rs` upon detection (maybe by detecting the magic byte / something defined in the specifications) or after getting a `ParseError` or `SchemaError` returned from the `Decoder`. I'm personally most familiar with schema registry (Kafka) use cases. We can definitely add utility functions to assist with this as needed and/or make the `Reader` and `Decoder` configurable to attempt to detect a new header definition record, but imo the responsibility should be on the caller since it's very use-case specific. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
