ahmedabu98 commented on PR #39156: URL: https://github.com/apache/beam/pull/39156#issuecomment-6353847615
Hey @atognolas, sorry for the slow turnaround on this. I think we still need to address the batch-with-autosharding case (see [comment above](https://github.com/apache/beam/pull/39156#discussion_r3494205093)). If a user has a batch pipeline and sets `autosharding=true`, we should honor that and use `GroupIntoBatches.withShardedKey()`. Your change overrides to use GroupByKey in all cases. Regarding @bvolpato's [comment](https://github.com/apache/beam/pull/39156#pullrequestreview-5003252448) above about letting the writer roll over to new files when it hits a byte size limit, I opened a PR some time ago to do exactly this but it's gone stale: https://github.com/apache/beam/pull/38930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
