Re: [I] Async avro reader optimization [arrow-rs]
ariel-miculas commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4419531896 > JSON decoder API: https://docs.rs/arrow-json/58.1.0/arrow_json/reader/index.html#async-usage The problem with the JSON decoder API is that it doesn't cover the datafusion use case of partitioning, i.e. handling inputs which end in the middle of JSON entries. The async Avro reader handles this case so it can be used as-is from datafusion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
mzabaluev commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4397986658 I agree it would be good to consistently split Avro decoding from I/O also in the batteries-included `AsyncAvroFileReader`. The pattern in arrow-rs, as I understand it, is to offload I/O operations onto a separate Tokio runtime provided e.g. in the object store reader's `with_runtime` method. The decoding is performed in the (also async) task of the stream's consumer, and it's up to the consumer to deal with potential CPU spikes there. As the FIXME [comment in #9632]( https://github.com/apache/arrow-rs/pull/9632/changes#diff-69d30880a02f73ac84bb04658fadea10605bea718a6efe35154fb4ff9457fa5aR102) says, we have departed from this pattern, but this can be improved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
alamb commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4316887132 > [@alamb](https://github.com/alamb) any concerns about the proposed changes or about the underlying issue that we're trying to solve? I guess what I am saying is that if we removed the IO from the Avro reader then clients that wanted more direct control could handle the IO themselves and push it to the decoder. example APIs I have in mind are: 1. ParquetPushDecoder: https://docs.rs/parquet/58.0.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html 2. JSON decoder API: https://docs.rs/arrow-json/58.1.0/arrow_json/reader/index.html#async-usage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
ariel-miculas commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4268677228 @alamb any concerns about the proposed changes or about the underlying issue that we're trying to solve? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
ariel-miculas commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4242773815 > rather than reading the entire requested file range in memory at once @alamb Note that the above is also a bug in itself: if there's not enough memory to read the entire file range, the existing AsyncFileReader will crash. So https://github.com/apache/arrow-rs/pull/9632 is not only about performance optimization, but it also fixes an important issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
mzabaluev commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4224694535 I don't know if the description is insufficient, but #9632 introduces an optimization: rather than reading the entire requested file range in memory at once and then parsing the file, the reader can request a stream and parse one chunk at a time as the file reader yields them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
alamb commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4214201622 Shall we close this ticket then? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Async avro reader optimization [arrow-rs]
mzabaluev commented on issue #9668: URL: https://github.com/apache/arrow-rs/issues/9668#issuecomment-4204778127 > Note we have also been trying to separate the IO from the decoding in Parquet -- see https://docs.rs/parquet/58.0.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html > > Perhaps we could move Avro to that model too rather than implementing the async stuff first There is already a push decoder implementation shared by the sync and async reader implementations. The differences and duplication mainly arise from the I/O model: for the sync reader it's stictly sequential, while the async reader is designed around ranged GET requests provided by `object_store`, with an abstraction to also support generic `AsyncRead + AsyncSeek`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
