Sunyue commented on issue #278: URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2735518194
> ng with repeated columns a `SeekToRow` doesn't make so much sense with the low level APIs (distinguishing rows vs values is not easy for skipping within a page). > I started implementing this a bit and realized that when you start dealing with repeated columns a `SeekToRow` doesn't make so much sense with the low level APIs (distinguishing rows vs values is not easy for skipping within a page). > > It makes more sense with the Arrow column and record readers in the `pqarrow` packages. What do you think? Hi @zeroshade I have similar use case and I came cross this issue. Like you mentioned here, I find it makes more sense to enhance `pqarrow` packages, e.g. enhance `RowGroupReader.ReadTable` to something like `func (rgr RowGroupReader) ReadTable(ctx context.Context, colIndices []int, offset, numOfRecords int) (arrow.Table, error)` This way we can read partial of the row group into memory, which makes it more memory efficient. Do you think it would be possible? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
