JkRuizGlob commented on issue #24906: URL: https://github.com/apache/beam/issues/24906#issuecomment-1406918263
I'm having basically the same issue also using Python 3.8 and Apache Beam 2.44.0, but in my case, I'm using Dataflow for running the beam pipeline. Also, I got to the same place than @lydian about the solution being to upgrade dill to 0.3.3 (https://github.com/uqfoundation/dill/issues/318) but it cannot be manually set since `apache-beam 2.44.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.3 which is incompatible` Previously I was using Python 3.7 with Apache Beam 2.36.0 and it was working fine but I need to upgrade the Python version so I went ahead and also updated the Apache Beam version to the latest one. I saw that you guys already had updated the dill version on the PR https://github.com/apache/beam/pull/17669 but then it was rolled back with this other one https://github.com/apache/beam/pull/21898 because there were some tests failing. Is there any ETA on when that dill version will be upgraded back again? I cannot downgrade the Python version to make it work, is there any other workaround for avoiding this error? ``` Error message from worker: Traceback (most recent call last): File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/internal/dill_pickler.py", line 285, in loads return dill.loads(s) File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) TypeError: an integer is required (got type bytes) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 287, in _execute response = task() File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 360, in <lambda> lambda: self.create_worker().do_instruction(request), request) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 596, in do_instruction return getattr(self, request_type)( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 627, in process_bundle bundle_processor = self.bundle_processor_cache.get( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 459, in get processor = bundle_processor.BundleProcessor( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 871, in __init__ self.ops = self.create_execution_tree(self.process_bundle_descriptor) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 924, in create_execution_tree return collections.OrderedDict([( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 927, in <listcomp> get_operation(transform_id))) for transform_id in sorted( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 814, in wrapper result = cache[args] = func(*args) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in get_operation transform_consumers = { File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 907, in <dictcomp> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]] File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 907, in <listcomp> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]] File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 814, in wrapper result = cache[args] = func(*args) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in get_operation transform_consumers = { File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 907, in <dictcomp> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]] File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 907, in <listcomp> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]] File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 814, in wrapper result = cache[args] = func(*args) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 906, in get_operation transform_consumers = { File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 907, in <dictcomp> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]] File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 907, in <listcomp> tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]] File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 814, in wrapper result = cache[args] = func(*args) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 911, in get_operation return transform_factory.create_operation( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1206, in create_operation return creator(self, transform_id, transform_proto, payload, consumers) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1463, in create_split_and_size_restrictions return _create_sdf_operation(SplitAndSizeRestrictions, *args) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1525, in _create_sdf_operation dofn_data = pickler.loads(parameter.do_fn.payload) File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/internal/pickler.py", line 51, in loads return desired_pickle_lib.loads( File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/internal/dill_pickler.py", line 289, in loads return dill.loads(s) File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) TypeError: an integer is required (got type bytes) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
