HippoBaro commented on PR #9697:
URL: https://github.com/apache/arrow-rs/pull/9697#issuecomment-4254648038
@alamb from the above PR you linked to:
> I also think we may need to adjust the Parquet PushDecoder somehow to
communicate better its internal boundaries (e.g. so we can easily / efficiently
pick off RowGroups without recreating it)
I see what you mean. Exposing internal boundaries to the caller is in
tension with the IO decoupling I'm pursuing here (ideally the decoder shouldn't
dictate how data is fetched or partitioned.) That said, take a look at the
`RetentionSet` introduced in c02b044: if work is split across consumer threads,
each could own its own `PushBuffers` with a `RetentionSet` scoped to its row
groups. The IO layer pushes the same data to all of them, and each decoder
admits only what it needs. No boundary negotiation required.
:2c:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
