| Hi Beam Community, I am running a Dataflow pipeline using the Python SDK. I am doing do some ETL processing on my data and then write the output into Big Query. When I try to write into Big Query I get below error in Dataflow job. However when running this pipeline from my local on DirectRunner the same code runs successfully and data is written into Big Query. "Clients have non-trivial state that is local and unpickleable.", _pickle.PicklingError: Pickling client objects is explicitly not supported. Clients have non-trivial state that is local and unpickleable. I have added the full traceback in the attached file. I am trying to write data into Big Query as below:- write_delivered_orders = (delivered_orders | "ConvertDeliveredToJSON" >> beam.Map(to_json) | "WriteDeliveredOrders" >> beam.io.WriteToBigQuery( delivered_order_table_spec, schema=table_schema, create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED, write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, additional_bq_parameters={'timePartitioning': {'type': 'DAY'}} ) ) Has anyone encountered this error, then can you please help me to understand and resolve it. Thanks in Advance. Thanks & Regards Rajnil Guha |
Traceback (most recent call last):
File "batch_pipeline.py", line 127, in <module>
additional_bq_parameters={'timePartitioning': {'type': 'DAY'}}
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/apache_beam/pipeline.py",
line 555, in __exit__
self.result = self.run()
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/apache_beam/pipeline.py",
line 521, in run
allow_proto_holders=True).run(False)
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/apache_beam/pipeline.py",
line 531, in run
pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/apache_beam/internal/pickler.py",
line 300, in dump_session
dill.dump_session(file_path)
File "/home/rio_guha94/beam_env/lib/python3.7/site-packages/dill/_dill.py",
line 351, in dump_session
pickler.dump(main)
File "/home/rio_guha94/beam_env/lib/python3.7/site-packages/dill/_dill.py",
line 445, in dump
StockPickler.dump(self, obj)
File "/usr/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/apache_beam/internal/pickler.py",
line 172, in save_module
return old_save_module(pickler, obj)
File "/home/rio_guha94/beam_env/lib/python3.7/site-packages/dill/_dill.py",
line 1295, in save_module
state=_main_dict)
File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/apache_beam/internal/pickler.py",
line 220, in new_save_module_dict
return old_save_module_dict(pickler, obj)
File "/home/rio_guha94/beam_env/lib/python3.7/site-packages/dill/_dill.py",
line 912, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/lib/python3.7/pickle.py", line 524, in save
rv = reduce(self.proto)
File
"/home/rio_guha94/beam_env/lib/python3.7/site-packages/google/cloud/client.py",
line 167, in __getstate__
"Clients have non-trivial state that is local and unpickleable.",
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
