alamb commented on PR #23492: URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5204925346
> My intuition is to have an IO pipeline that uses memory, row counts and request concurrency to tune work, and feeds into CPU decoding. But it’s the decoding that determines _what_ data to fetch, either known or speculative (the row filter case) and that owns the data in flight (so it can do things like discard speculative data that proves non useful). The IO pipeline asks the decoder “hey i’m going to make my next 1-4MB request, what do you want in there?” and the decoder + some IO policy decide what bytes they want. Some other potentially related work is Lance's https://github.com/lance-format/lance/blob/71c4aa2174971e98acb7e256fde1e1589024f5bc/rust/lance-encoding/src/decoder.rs#L424-L445 Here is their trait for the IO interface: https://github.com/lance-format/lance/blob/71c4aa2174971e98acb7e256fde1e1589024f5bc/rust/lance-encoding/src/lib.rs#L41 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
