adriangb commented on PR #23492: URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5197816153
Sorry I haven’t replied here. I hope to have time tomorrow to look in more detail. I’m not sure what the right architecture is yet, but my intuition is that: - It should not be based on row groups or row numbers/batch sizes. Consider an int32 column vs a large string column: you can probably prefetch 1M rows of the int32 column each IO operation, you may only want a couple rows of the large string column. - Like DuckDBs design it should be based on a decoupled async IO pipeline so we can make sure we keep the CPU and IO saturated - It needs to account for vastly different IO profiles (SDD vs S3) My intuition is to have an IO pipeline that uses memory, row counts and request concurrency to tune work, and feeds into CPU decoding. But it’s the decoding that determines *what* data to fetch, either known or speculative (the row filter case) and that owns the data in flight (so it can do things like discard speculative data that proves non useful). The IO pipeline asks the decoder “hey i’m going to make my next 1-4MB request, what do you want in there?” and the decoder + some IO policy decide what bytes they want. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
