claudevdm commented on code in PR #36271: URL: https://github.com/apache/beam/pull/36271#discussion_r2392281246
########## sdks/python/apache_beam/internal/cloudpickle_pickler.py: ########## @@ -196,12 +196,35 @@ def _lock_reducer(obj): def dump_session(file_path): - # It is possible to dump session with cloudpickle. However, since references - # are saved it should not be necessary. See https://s.apache.org/beam-picklers - pass + # Since References are saved (https://s.apache.org/beam-picklers), we only + # dump supported Beam Registries (currently only logical type registry) + from apache_beam.typehints import schemas + from apache_beam.coders import typecoders Review Comment: Does it matter if the coder is registered on the worker though? My understanding is `coders.registry.register_coder(JdbcTestRow, coders.RowCoder)` just maps any transforms that use JdbcTestRow to use RowCoder during pipeline construction. Then the RowCoder is pickled as part of the pipeline proto, and the worker doesnt actually need the coder registry, it just uses the pickled coder. Am I missing something? Can you reproduce some sort of error in the case where coder registry is not correct on worker? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
