alamb commented on issue #8000: URL: https://github.com/apache/arrow-rs/issues/8000#issuecomment-3191950291
> Very cool [@alamb](https://github.com/alamb)! > > > 2. Flexibility in how the reader intermixes IO and CPU work, so that users can better control the interleaving of IO and CPU work during decode. > > It seems somewhat similar of the work that was done on Lance version 2, where the IO scheduling was decoupled from the actual decoding? https://github.com/lancedb/lance/blob/main/rust/lance-encoding/src/decoder.rs#L108-L139 Yes indeed -- @westonpace describes how it works a bit more in detail here: https://github.com/apache/arrow-rs/issues/7983#issuecomment-3164659936 What I am planning for the parquet reader in arrow-rs is not the sophisticated IO scheduling that Lance has, but rather a decoder that people could build similarly sophisticated IO scheduling in their own systems if desired. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
