sharan-malyala opened a new pull request, #35110: URL: https://github.com/apache/beam/pull/35110
Setting max_writers_per_bundle=0 in writeToJson() so that it is passed to WriteToFiles transform. This allows the PCollections to be processed as part of [WriteShardedRecordsFn](https://github.com/apache/beam/blob/82ef9d81066226833aaf8c1b30aae11e46e62771/sdks/python/apache_beam/io/fileio.py#L621) step which allows sharding based on the user specified num_shards or default shards. It can create larger files and increase parallelism. Currently, writes happen via [WriteUnshardedRecordsFn](https://github.com/apache/beam/blob/82ef9d81066226833aaf8c1b30aae11e46e62771/sdks/python/apache_beam/io/fileio.py#L609) step which creates a single key because the sharding is dependent on destination and destination is not available to pass via writeToJson. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
