mzabaluev commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4397986658
I agree it would be good to consistently split Avro decoding from I/O also in the batteries-included `AsyncAvroFileReader`. The pattern in arrow-rs, as I understand it, is to offload I/O operations onto a separate Tokio runtime provided e.g. in the object store reader's `with_runtime` method. The decoding is performed in the (also async) task of the stream's consumer, and it's up to the consumer to deal with potential CPU spikes there. As the FIXME [comment in #9632]( https://github.com/apache/arrow-rs/pull/9632/changes#diff-69d30880a02f73ac84bb04658fadea10605bea718a6efe35154fb4ff9457fa5aR102) says, we have departed from this pattern, but this can be improved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
