The fact that you have cloudpickle==1.2.2 further confirms that you may be hitting the same error as https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype .
Could you try to start over with a clean virtual environment? On Tue, Feb 4, 2020 at 11:46 AM Alan Krumholz <alan.krumh...@betterup.co> wrote: > Hi Valentyn, > > Here is my pip freeze on my machine (note that the error is in dataflow, > the job runs fine in my machine) > > ansiwrap==0.8.4 > apache-beam==2.19.0 > arrow==0.15.5 > asn1crypto==1.3.0 > astroid==2.3.3 > astropy==3.2.3 > attrs==19.3.0 > avro-python3==1.9.1 > azure-common==1.1.24 > azure-storage-blob==2.1.0 > azure-storage-common==2.1.0 > backcall==0.1.0 > bcolz==1.2.1 > binaryornot==0.4.4 > bleach==3.1.0 > boto3==1.11.9 > botocore==1.14.9 > cachetools==3.1.1 > certifi==2019.11.28 > cffi==1.13.2 > chardet==3.0.4 > Click==7.0 > cloudpickle==1.2.2 > colorama==0.4.3 > configparser==4.0.2 > confuse==1.0.0 > cookiecutter==1.7.0 > crcmod==1.7 > cryptography==2.8 > cycler==0.10.0 > daal==2019.0 > datalab==1.1.5 > decorator==4.4.1 > defusedxml==0.6.0 > dill==0.3.1.1 > distro==1.0.1 > docker==4.1.0 > docopt==0.6.2 > docutils==0.15.2 > entrypoints==0.3 > enum34==1.1.6 > fairing==0.5.3 > fastavro==0.21.24 > fasteners==0.15 > fsspec==0.6.2 > future==0.18.2 > gcsfs==0.6.0 > gitdb2==2.0.6 > GitPython==3.0.5 > google-api-core==1.16.0 > google-api-python-client==1.7.11 > google-apitools==0.5.28 > google-auth==1.11.0 > google-auth-httplib2==0.0.3 > google-auth-oauthlib==0.4.1 > google-cloud-bigquery==1.17.1 > google-cloud-bigtable==1.0.0 > google-cloud-core==1.2.0 > google-cloud-dataproc==0.6.1 > google-cloud-datastore==1.7.4 > google-cloud-language==1.3.0 > google-cloud-logging==1.14.0 > google-cloud-monitoring==0.31.1 > google-cloud-pubsub==1.0.2 > google-cloud-secret-manager==0.1.1 > google-cloud-spanner==1.13.0 > google-cloud-storage==1.25.0 > google-cloud-translate==2.0.0 > google-compute-engine==20191210.0 > google-resumable-media==0.4.1 > googleapis-common-protos==1.51.0 > grpc-google-iam-v1==0.12.3 > grpcio==1.26.0 > h5py==2.10.0 > hdfs==2.5.8 > html5lib==1.0.1 > htmlmin==0.1.12 > httplib2==0.12.0 > icc-rt==2020.0.133 > idna==2.8 > ijson==2.6.1 > imageio==2.6.1 > importlib-metadata==1.4.0 > intel-numpy==1.15.1 > intel-openmp==2020.0.133 > intel-scikit-learn==0.19.2 > intel-scipy==1.1.0 > ipykernel==5.1.4 > ipython==7.9.0 > ipython-genutils==0.2.0 > ipython-sql==0.3.9 > ipywidgets==7.5.1 > isort==4.3.21 > jedi==0.16.0 > Jinja2==2.11.0 > jinja2-time==0.2.0 > jmespath==0.9.4 > joblib==0.14.1 > json5==0.8.5 > jsonschema==3.2.0 > jupyter==1.0.0 > jupyter-aihub-deploy-extension==0.1 > jupyter-client==5.3.4 > jupyter-console==6.1.0 > jupyter-contrib-core==0.3.3 > jupyter-contrib-nbextensions==0.5.1 > jupyter-core==4.6.1 > jupyter-highlight-selected-word==0.2.0 > jupyter-http-over-ws==0.0.7 > jupyter-latex-envs==1.4.6 > jupyter-nbextensions-configurator==0.4.1 > jupyterlab==1.2.6 > jupyterlab-git==0.9.0 > jupyterlab-server==1.0.6 > keyring==10.1 > keyrings.alt==1.3 > kiwisolver==1.1.0 > kubernetes==10.0.1 > lazy-object-proxy==1.4.3 > llvmlite==0.31.0 > lxml==4.4.2 > Markdown==3.1.1 > MarkupSafe==1.1.1 > matplotlib==3.0.3 > mccabe==0.6.1 > missingno==0.4.2 > mistune==0.8.4 > mkl==2019.0 > mkl-fft==1.0.6 > mkl-random==1.0.1.1 > mock==2.0.0 > monotonic==1.5 > more-itertools==8.1.0 > nbconvert==5.6.1 > nbdime==1.1.0 > nbformat==5.0.4 > networkx==2.4 > nltk==3.4.5 > notebook==6.0.3 > numba==0.47.0 > numpy==1.15.1 > oauth2client==3.0.0 > oauthlib==3.1.0 > opencv-python==4.1.2.30 > oscrypto==1.2.0 > packaging==20.1 > pandas==0.25.3 > pandas-profiling==1.4.0 > pandocfilters==1.4.2 > papermill==1.2.1 > parso==0.6.0 > pathlib2==2.3.5 > pbr==5.4.4 > pexpect==4.8.0 > phik==0.9.8 > pickleshare==0.7.5 > Pillow-SIMD==6.2.2.post1 > pipdeptree==0.13.2 > plotly==4.5.0 > pluggy==0.13.1 > poyo==0.5.0 > prettytable==0.7.2 > prometheus-client==0.7.1 > prompt-toolkit==2.0.10 > protobuf==3.11.2 > psutil==5.6.7 > ptyprocess==0.6.0 > py==1.8.1 > pyarrow==0.15.1 > pyasn1==0.4.8 > pyasn1-modules==0.2.8 > pycparser==2.19 > pycrypto==2.6.1 > pycryptodomex==3.9.6 > pycurl==7.43.0 > pydaal==2019.0.0.20180713 > pydot==1.4.1 > Pygments==2.5.2 > pygobject==3.22.0 > PyJWT==1.7.1 > pylint==2.4.4 > pymongo==3.10.1 > pyOpenSSL==19.1.0 > pyparsing==2.4.6 > pyrsistent==0.15.7 > pytest==5.3.4 > pytest-pylint==0.14.1 > python-apt==1.4.1 > python-dateutil==2.8.1 > pytz==2019.3 > PyWavelets==1.1.1 > pyxdg==0.25 > PyYAML==5.3 > pyzmq==18.1.1 > qtconsole==4.6.0 > requests==2.22.0 > requests-oauthlib==1.3.0 > retrying==1.3.3 > rsa==4.0 > s3transfer==0.3.2 > scikit-image==0.15.0 > scikit-learn==0.19.2 > scipy==1.1.0 > seaborn==0.9.1 > SecretStorage==2.3.1 > Send2Trash==1.5.0 > simplegeneric==0.8.1 > six==1.14.0 > smmap2==2.0.5 > snowflake-connector-python==2.2.0 > SQLAlchemy==1.3.13 > sqlparse==0.3.0 > tbb==2019.0 > tbb4py==2019.0 > tenacity==6.0.0 > terminado==0.8.3 > testpath==0.4.4 > textwrap3==0.9.2 > tornado==5.1.1 > tqdm==4.42.0 > traitlets==4.3.3 > typed-ast==1.4.1 > typing==3.7.4.1 > typing-extensions==3.7.4.1 > unattended-upgrades==0.1 > uritemplate==3.0.1 > urllib3==1.24.2 > virtualenv==16.7.9 > wcwidth==0.1.8 > webencodings==0.5.1 > websocket-client==0.57.0 > Werkzeug==0.16.1 > whichcraft==0.6.1 > widgetsnbextension==3.5.1 > wrapt==1.11.2 > zipp==1.1.0 > > > On Tue, Feb 4, 2020 at 11:33 AM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> It don't think there is a mismatch between dill versions here, but >> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >> mentions >> a similar error and may be related. What is the output of pip freeze on >> your machine (or better: pip install pipdeptree; pipdeptree)? >> >> >> On Tue, Feb 4, 2020 at 11:22 AM Alan Krumholz <alan.krumh...@betterup.co> >> wrote: >> >>> Here is a test job that sometimes fails and sometimes doesn't (but most >>> times do)..... >>> There seems to be something stochastic that causes this as after several >>> tests a couple of them did succeed.... >>> >>> >>> def test_error( >>> bq_table: str) -> str: >>> >>> import apache_beam as beam >>> from apache_beam.options.pipeline_options import PipelineOptions >>> >>> class GenData(beam.DoFn): >>> def process(self, _): >>> for _ in range (20000): >>> yield {'a':1,'b':2} >>> >>> >>> def get_bigquery_schema(): >>> from apache_beam.io.gcp.internal.clients import bigquery >>> >>> table_schema = bigquery.TableSchema() >>> columns = [ >>> ["a","integer","nullable"], >>> ["b","integer","nullable"] >>> ] >>> >>> for column in columns: >>> column_schema = bigquery.TableFieldSchema() >>> column_schema.name = column[0] >>> column_schema.type = column[1] >>> column_schema.mode = column[2] >>> table_schema.fields.append(column_schema) >>> >>> return table_schema >>> >>> pipeline = beam.Pipeline(options=PipelineOptions( >>> project='my-project', >>> temp_location = 'gs://my-bucket/temp', >>> staging_location = 'gs://my-bucket/staging', >>> runner='DataflowRunner' >>> )) >>> #pipeline = beam.Pipeline() >>> >>> ( >>> pipeline >>> | 'Empty start' >> beam.Create(['']) >>> | 'Generate Data' >> beam.ParDo(GenData()) >>> #| 'print' >> beam.Map(print) >>> | 'Write to BigQuery' >> beam.io.WriteToBigQuery( >>> project=bq_table.split(':')[0], >>> dataset=bq_table.split(':')[1].split('.')[0], >>> table=bq_table.split(':')[1].split('.')[1], >>> schema=get_bigquery_schema(), >>> >>> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED, >>> >>> write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE) >>> ) >>> >>> result = pipeline.run() >>> result.wait_until_finish() >>> >>> return True >>> >>> test_error( >>> bq_table = 'my-project:my_dataset.my_table' >>> ) >>> >>> On Tue, Feb 4, 2020 at 10:04 AM Alan Krumholz <alan.krumh...@betterup.co> >>> wrote: >>> >>>> I tried breaking apart my pipeline. Seems the step that breaks it is: >>>> beam.io.WriteToBigQuery >>>> >>>> Let me see if I can create a self contained example that breaks to >>>> share with you >>>> >>>> Thanks! >>>> >>>> On Tue, Feb 4, 2020 at 9:53 AM Pablo Estrada <pabl...@google.com> >>>> wrote: >>>> >>>>> Hm that's odd. No changes to the pipeline? Are you able to share some >>>>> of the code? >>>>> >>>>> +Udi Meiri <eh...@google.com> do you have any idea what could be >>>>> going on here? >>>>> >>>>> On Tue, Feb 4, 2020 at 9:25 AM Alan Krumholz < >>>>> alan.krumh...@betterup.co> wrote: >>>>> >>>>>> Hi Pablo, >>>>>> This is strange... it doesn't seem to be the last beam release as >>>>>> last night it was already using 2.19.0 I wonder if it was some release >>>>>> from >>>>>> the DataFlow team (not beam related): >>>>>> Job typeBatch >>>>>> Job status Succeeded >>>>>> SDK version >>>>>> Apache Beam Python 3.5 SDK 2.19.0 >>>>>> Region >>>>>> us-central1 >>>>>> Start timeFebruary 3, 2020 at 9:28:35 PM GMT-8 >>>>>> Elapsed time5 min 11 sec >>>>>> >>>>>> On Tue, Feb 4, 2020 at 9:15 AM Pablo Estrada <pabl...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Alan, >>>>>>> could it be that you're picking up the new Apache Beam 2.19.0 >>>>>>> release? Could you try depending on beam 2.18.0 to see if the issue >>>>>>> surfaces when using the new release? >>>>>>> >>>>>>> If something was working and no longer works, it sounds like a bug. >>>>>>> This may have to do with how we pickle (dill / cloudpickle) - see this >>>>>>> question >>>>>>> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >>>>>>> Best >>>>>>> -P. >>>>>>> >>>>>>> On Tue, Feb 4, 2020 at 6:22 AM Alan Krumholz < >>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I was running a dataflow job in GCP last night and it was running >>>>>>>> fine. >>>>>>>> This morning this same exact job is failing with the following >>>>>>>> error: >>>>>>>> >>>>>>>> Error message from worker: Traceback (most recent call last): File >>>>>>>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>>>>>>> line 286, in loads return dill.loads(s) File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in >>>>>>>> loads >>>>>>>> return load(file, ignore, **kwds) File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in >>>>>>>> load >>>>>>>> return Unpickler(file, ignore=ignore, **kwds).load() File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in >>>>>>>> load >>>>>>>> obj = StockUnpickler.load(self) File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >>>>>>>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' During >>>>>>>> handling of the above exception, another exception occurred: Traceback >>>>>>>> (most recent call last): File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", >>>>>>>> line 648, in do_work work_executor.execute() File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", >>>>>>>> line >>>>>>>> 176, in execute op.start() File >>>>>>>> "apache_beam/runners/worker/operations.py", >>>>>>>> line 649, in apache_beam.runners.worker.operations.DoOperation.start >>>>>>>> File >>>>>>>> "apache_beam/runners/worker/operations.py", line 651, in >>>>>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>>>>> "apache_beam/runners/worker/operations.py", line 652, in >>>>>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>>>>> "apache_beam/runners/worker/operations.py", line 261, in >>>>>>>> apache_beam.runners.worker.operations.Operation.start File >>>>>>>> "apache_beam/runners/worker/operations.py", line 266, in >>>>>>>> apache_beam.runners.worker.operations.Operation.start File >>>>>>>> "apache_beam/runners/worker/operations.py", line 597, in >>>>>>>> apache_beam.runners.worker.operations.DoOperation.setup File >>>>>>>> "apache_beam/runners/worker/operations.py", line 602, in >>>>>>>> apache_beam.runners.worker.operations.DoOperation.setup File >>>>>>>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>>>>>>> line 290, in loads return dill.loads(s) File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in >>>>>>>> loads >>>>>>>> return load(file, ignore, **kwds) File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in >>>>>>>> load >>>>>>>> return Unpickler(file, ignore=ignore, **kwds).load() File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in >>>>>>>> load >>>>>>>> obj = StockUnpickler.load(self) File >>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >>>>>>>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' >>>>>>>> >>>>>>>> >>>>>>>> If I use a local runner it still runs fine. >>>>>>>> Anyone else experiencing something similar today? (or know how to >>>>>>>> fix this?) >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>
