Abacn commented on code in PR #36271: URL: https://github.com/apache/beam/pull/36271#discussion_r2392212616
########## sdks/python/apache_beam/internal/cloudpickle_pickler.py: ########## @@ -196,12 +196,35 @@ def _lock_reducer(obj): def dump_session(file_path): - # It is possible to dump session with cloudpickle. However, since references - # are saved it should not be necessary. See https://s.apache.org/beam-picklers - pass + # Since References are saved (https://s.apache.org/beam-picklers), we only + # dump supported Beam Registries (currently only logical type registry) + from apache_beam.typehints import schemas + from apache_beam.coders import typecoders Review Comment: It's a related issue. I made a minimum example pipeline based on https://github.com/apache/beam/blob/bb340c2f66ac8730334160d6ed5ecd18822d059d/sdks/python/apache_beam/io/external/xlang_jdbcio_it_test.py in particular, in L67 there it registered a coder > coders.registry.register_coder(JdbcTestRow, coders.RowCoder) I checked with dill and save main session, this mapping is registered in coder registry on worker. But for cloudpickle, it is no longer registered on worker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
