perfect! thank you! On Fri, Feb 7, 2020 at 10:54 AM Valentyn Tymofieiev <valen...@google.com> wrote:
> Thanks for your feedback. We expect that this issue will be fixed in > cloudpickle==1.3.0. Per [1], this release may be available next week. > > After that you can install the fixed version of cloudpickle until the AI > notebook image picks up the new version. > > [1] https://github.com/cloudpipe/cloudpickle/pull/337 > > On Tue, Feb 4, 2020 at 12:44 PM Alan Krumholz <alan.krumh...@betterup.co> > wrote: > >> Seems like the image we use in KFP to orchestrate the job has >> cloudpickle==0.8.1 >> and that one doesn't seem to cause issues. >> I think I'm unblock for now but I'm sure I won't be the last one to try >> to do this using GCP managed notebooks :( >> >> Thanks for all the help! >> >> >> On Tue, Feb 4, 2020 at 12:24 PM Alan Krumholz <alan.krumh...@betterup.co> >> wrote: >> >>> I'm using a managed notebook instance from GCP >>> It seems those already come with cloudpickle==1.2.2 as soon as you >>> provision it. apache-beam[gcp] will then install dill==0.3.1.1 I'm >>> going to try to uninstall cloudpickle before installing apache-beam and see >>> if this fixes the problem >>> >>> Thank you >>> >>> On Tue, Feb 4, 2020 at 11:54 AM Valentyn Tymofieiev <valen...@google.com> >>> wrote: >>> >>>> The fact that you have cloudpickle==1.2.2 further confirms that you >>>> may be hitting the same error as >>>> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >>>> . >>>> >>>> Could you try to start over with a clean virtual environment? >>>> >>>> On Tue, Feb 4, 2020 at 11:46 AM Alan Krumholz < >>>> alan.krumh...@betterup.co> wrote: >>>> >>>>> Hi Valentyn, >>>>> >>>>> Here is my pip freeze on my machine (note that the error is in >>>>> dataflow, the job runs fine in my machine) >>>>> >>>>> ansiwrap==0.8.4 >>>>> apache-beam==2.19.0 >>>>> arrow==0.15.5 >>>>> asn1crypto==1.3.0 >>>>> astroid==2.3.3 >>>>> astropy==3.2.3 >>>>> attrs==19.3.0 >>>>> avro-python3==1.9.1 >>>>> azure-common==1.1.24 >>>>> azure-storage-blob==2.1.0 >>>>> azure-storage-common==2.1.0 >>>>> backcall==0.1.0 >>>>> bcolz==1.2.1 >>>>> binaryornot==0.4.4 >>>>> bleach==3.1.0 >>>>> boto3==1.11.9 >>>>> botocore==1.14.9 >>>>> cachetools==3.1.1 >>>>> certifi==2019.11.28 >>>>> cffi==1.13.2 >>>>> chardet==3.0.4 >>>>> Click==7.0 >>>>> cloudpickle==1.2.2 >>>>> colorama==0.4.3 >>>>> configparser==4.0.2 >>>>> confuse==1.0.0 >>>>> cookiecutter==1.7.0 >>>>> crcmod==1.7 >>>>> cryptography==2.8 >>>>> cycler==0.10.0 >>>>> daal==2019.0 >>>>> datalab==1.1.5 >>>>> decorator==4.4.1 >>>>> defusedxml==0.6.0 >>>>> dill==0.3.1.1 >>>>> distro==1.0.1 >>>>> docker==4.1.0 >>>>> docopt==0.6.2 >>>>> docutils==0.15.2 >>>>> entrypoints==0.3 >>>>> enum34==1.1.6 >>>>> fairing==0.5.3 >>>>> fastavro==0.21.24 >>>>> fasteners==0.15 >>>>> fsspec==0.6.2 >>>>> future==0.18.2 >>>>> gcsfs==0.6.0 >>>>> gitdb2==2.0.6 >>>>> GitPython==3.0.5 >>>>> google-api-core==1.16.0 >>>>> google-api-python-client==1.7.11 >>>>> google-apitools==0.5.28 >>>>> google-auth==1.11.0 >>>>> google-auth-httplib2==0.0.3 >>>>> google-auth-oauthlib==0.4.1 >>>>> google-cloud-bigquery==1.17.1 >>>>> google-cloud-bigtable==1.0.0 >>>>> google-cloud-core==1.2.0 >>>>> google-cloud-dataproc==0.6.1 >>>>> google-cloud-datastore==1.7.4 >>>>> google-cloud-language==1.3.0 >>>>> google-cloud-logging==1.14.0 >>>>> google-cloud-monitoring==0.31.1 >>>>> google-cloud-pubsub==1.0.2 >>>>> google-cloud-secret-manager==0.1.1 >>>>> google-cloud-spanner==1.13.0 >>>>> google-cloud-storage==1.25.0 >>>>> google-cloud-translate==2.0.0 >>>>> google-compute-engine==20191210.0 >>>>> google-resumable-media==0.4.1 >>>>> googleapis-common-protos==1.51.0 >>>>> grpc-google-iam-v1==0.12.3 >>>>> grpcio==1.26.0 >>>>> h5py==2.10.0 >>>>> hdfs==2.5.8 >>>>> html5lib==1.0.1 >>>>> htmlmin==0.1.12 >>>>> httplib2==0.12.0 >>>>> icc-rt==2020.0.133 >>>>> idna==2.8 >>>>> ijson==2.6.1 >>>>> imageio==2.6.1 >>>>> importlib-metadata==1.4.0 >>>>> intel-numpy==1.15.1 >>>>> intel-openmp==2020.0.133 >>>>> intel-scikit-learn==0.19.2 >>>>> intel-scipy==1.1.0 >>>>> ipykernel==5.1.4 >>>>> ipython==7.9.0 >>>>> ipython-genutils==0.2.0 >>>>> ipython-sql==0.3.9 >>>>> ipywidgets==7.5.1 >>>>> isort==4.3.21 >>>>> jedi==0.16.0 >>>>> Jinja2==2.11.0 >>>>> jinja2-time==0.2.0 >>>>> jmespath==0.9.4 >>>>> joblib==0.14.1 >>>>> json5==0.8.5 >>>>> jsonschema==3.2.0 >>>>> jupyter==1.0.0 >>>>> jupyter-aihub-deploy-extension==0.1 >>>>> jupyter-client==5.3.4 >>>>> jupyter-console==6.1.0 >>>>> jupyter-contrib-core==0.3.3 >>>>> jupyter-contrib-nbextensions==0.5.1 >>>>> jupyter-core==4.6.1 >>>>> jupyter-highlight-selected-word==0.2.0 >>>>> jupyter-http-over-ws==0.0.7 >>>>> jupyter-latex-envs==1.4.6 >>>>> jupyter-nbextensions-configurator==0.4.1 >>>>> jupyterlab==1.2.6 >>>>> jupyterlab-git==0.9.0 >>>>> jupyterlab-server==1.0.6 >>>>> keyring==10.1 >>>>> keyrings.alt==1.3 >>>>> kiwisolver==1.1.0 >>>>> kubernetes==10.0.1 >>>>> lazy-object-proxy==1.4.3 >>>>> llvmlite==0.31.0 >>>>> lxml==4.4.2 >>>>> Markdown==3.1.1 >>>>> MarkupSafe==1.1.1 >>>>> matplotlib==3.0.3 >>>>> mccabe==0.6.1 >>>>> missingno==0.4.2 >>>>> mistune==0.8.4 >>>>> mkl==2019.0 >>>>> mkl-fft==1.0.6 >>>>> mkl-random==1.0.1.1 >>>>> mock==2.0.0 >>>>> monotonic==1.5 >>>>> more-itertools==8.1.0 >>>>> nbconvert==5.6.1 >>>>> nbdime==1.1.0 >>>>> nbformat==5.0.4 >>>>> networkx==2.4 >>>>> nltk==3.4.5 >>>>> notebook==6.0.3 >>>>> numba==0.47.0 >>>>> numpy==1.15.1 >>>>> oauth2client==3.0.0 >>>>> oauthlib==3.1.0 >>>>> opencv-python==4.1.2.30 >>>>> oscrypto==1.2.0 >>>>> packaging==20.1 >>>>> pandas==0.25.3 >>>>> pandas-profiling==1.4.0 >>>>> pandocfilters==1.4.2 >>>>> papermill==1.2.1 >>>>> parso==0.6.0 >>>>> pathlib2==2.3.5 >>>>> pbr==5.4.4 >>>>> pexpect==4.8.0 >>>>> phik==0.9.8 >>>>> pickleshare==0.7.5 >>>>> Pillow-SIMD==6.2.2.post1 >>>>> pipdeptree==0.13.2 >>>>> plotly==4.5.0 >>>>> pluggy==0.13.1 >>>>> poyo==0.5.0 >>>>> prettytable==0.7.2 >>>>> prometheus-client==0.7.1 >>>>> prompt-toolkit==2.0.10 >>>>> protobuf==3.11.2 >>>>> psutil==5.6.7 >>>>> ptyprocess==0.6.0 >>>>> py==1.8.1 >>>>> pyarrow==0.15.1 >>>>> pyasn1==0.4.8 >>>>> pyasn1-modules==0.2.8 >>>>> pycparser==2.19 >>>>> pycrypto==2.6.1 >>>>> pycryptodomex==3.9.6 >>>>> pycurl==7.43.0 >>>>> pydaal==2019.0.0.20180713 >>>>> pydot==1.4.1 >>>>> Pygments==2.5.2 >>>>> pygobject==3.22.0 >>>>> PyJWT==1.7.1 >>>>> pylint==2.4.4 >>>>> pymongo==3.10.1 >>>>> pyOpenSSL==19.1.0 >>>>> pyparsing==2.4.6 >>>>> pyrsistent==0.15.7 >>>>> pytest==5.3.4 >>>>> pytest-pylint==0.14.1 >>>>> python-apt==1.4.1 >>>>> python-dateutil==2.8.1 >>>>> pytz==2019.3 >>>>> PyWavelets==1.1.1 >>>>> pyxdg==0.25 >>>>> PyYAML==5.3 >>>>> pyzmq==18.1.1 >>>>> qtconsole==4.6.0 >>>>> requests==2.22.0 >>>>> requests-oauthlib==1.3.0 >>>>> retrying==1.3.3 >>>>> rsa==4.0 >>>>> s3transfer==0.3.2 >>>>> scikit-image==0.15.0 >>>>> scikit-learn==0.19.2 >>>>> scipy==1.1.0 >>>>> seaborn==0.9.1 >>>>> SecretStorage==2.3.1 >>>>> Send2Trash==1.5.0 >>>>> simplegeneric==0.8.1 >>>>> six==1.14.0 >>>>> smmap2==2.0.5 >>>>> snowflake-connector-python==2.2.0 >>>>> SQLAlchemy==1.3.13 >>>>> sqlparse==0.3.0 >>>>> tbb==2019.0 >>>>> tbb4py==2019.0 >>>>> tenacity==6.0.0 >>>>> terminado==0.8.3 >>>>> testpath==0.4.4 >>>>> textwrap3==0.9.2 >>>>> tornado==5.1.1 >>>>> tqdm==4.42.0 >>>>> traitlets==4.3.3 >>>>> typed-ast==1.4.1 >>>>> typing==3.7.4.1 >>>>> typing-extensions==3.7.4.1 >>>>> unattended-upgrades==0.1 >>>>> uritemplate==3.0.1 >>>>> urllib3==1.24.2 >>>>> virtualenv==16.7.9 >>>>> wcwidth==0.1.8 >>>>> webencodings==0.5.1 >>>>> websocket-client==0.57.0 >>>>> Werkzeug==0.16.1 >>>>> whichcraft==0.6.1 >>>>> widgetsnbextension==3.5.1 >>>>> wrapt==1.11.2 >>>>> zipp==1.1.0 >>>>> >>>>> >>>>> On Tue, Feb 4, 2020 at 11:33 AM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> It don't think there is a mismatch between dill versions here, but >>>>>> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >>>>>> mentions >>>>>> a similar error and may be related. What is the output of pip freeze on >>>>>> your machine (or better: pip install pipdeptree; pipdeptree)? >>>>>> >>>>>> >>>>>> On Tue, Feb 4, 2020 at 11:22 AM Alan Krumholz < >>>>>> alan.krumh...@betterup.co> wrote: >>>>>> >>>>>>> Here is a test job that sometimes fails and sometimes doesn't (but >>>>>>> most times do)..... >>>>>>> There seems to be something stochastic that causes this as after >>>>>>> several tests a couple of them did succeed.... >>>>>>> >>>>>>> >>>>>>> def test_error( >>>>>>> bq_table: str) -> str: >>>>>>> >>>>>>> import apache_beam as beam >>>>>>> from apache_beam.options.pipeline_options import PipelineOptions >>>>>>> >>>>>>> class GenData(beam.DoFn): >>>>>>> def process(self, _): >>>>>>> for _ in range (20000): >>>>>>> yield {'a':1,'b':2} >>>>>>> >>>>>>> >>>>>>> def get_bigquery_schema(): >>>>>>> from apache_beam.io.gcp.internal.clients import bigquery >>>>>>> >>>>>>> table_schema = bigquery.TableSchema() >>>>>>> columns = [ >>>>>>> ["a","integer","nullable"], >>>>>>> ["b","integer","nullable"] >>>>>>> ] >>>>>>> >>>>>>> for column in columns: >>>>>>> column_schema = bigquery.TableFieldSchema() >>>>>>> column_schema.name = column[0] >>>>>>> column_schema.type = column[1] >>>>>>> column_schema.mode = column[2] >>>>>>> table_schema.fields.append(column_schema) >>>>>>> >>>>>>> return table_schema >>>>>>> >>>>>>> pipeline = beam.Pipeline(options=PipelineOptions( >>>>>>> project='my-project', >>>>>>> temp_location = 'gs://my-bucket/temp', >>>>>>> staging_location = 'gs://my-bucket/staging', >>>>>>> runner='DataflowRunner' >>>>>>> )) >>>>>>> #pipeline = beam.Pipeline() >>>>>>> >>>>>>> ( >>>>>>> pipeline >>>>>>> | 'Empty start' >> beam.Create(['']) >>>>>>> | 'Generate Data' >> beam.ParDo(GenData()) >>>>>>> #| 'print' >> beam.Map(print) >>>>>>> | 'Write to BigQuery' >> beam.io.WriteToBigQuery( >>>>>>> project=bq_table.split(':')[0], >>>>>>> dataset=bq_table.split(':')[1].split('.')[0], >>>>>>> table=bq_table.split(':')[1].split('.')[1], >>>>>>> schema=get_bigquery_schema(), >>>>>>> >>>>>>> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED, >>>>>>> >>>>>>> write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE) >>>>>>> ) >>>>>>> >>>>>>> result = pipeline.run() >>>>>>> result.wait_until_finish() >>>>>>> >>>>>>> return True >>>>>>> >>>>>>> test_error( >>>>>>> bq_table = 'my-project:my_dataset.my_table' >>>>>>> ) >>>>>>> >>>>>>> On Tue, Feb 4, 2020 at 10:04 AM Alan Krumholz < >>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>> >>>>>>>> I tried breaking apart my pipeline. Seems the step that breaks it >>>>>>>> is: >>>>>>>> beam.io.WriteToBigQuery >>>>>>>> >>>>>>>> Let me see if I can create a self contained example that breaks to >>>>>>>> share with you >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> On Tue, Feb 4, 2020 at 9:53 AM Pablo Estrada <pabl...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hm that's odd. No changes to the pipeline? Are you able to share >>>>>>>>> some of the code? >>>>>>>>> >>>>>>>>> +Udi Meiri <eh...@google.com> do you have any idea what could be >>>>>>>>> going on here? >>>>>>>>> >>>>>>>>> On Tue, Feb 4, 2020 at 9:25 AM Alan Krumholz < >>>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>>> >>>>>>>>>> Hi Pablo, >>>>>>>>>> This is strange... it doesn't seem to be the last beam release as >>>>>>>>>> last night it was already using 2.19.0 I wonder if it was some >>>>>>>>>> release from >>>>>>>>>> the DataFlow team (not beam related): >>>>>>>>>> Job typeBatch >>>>>>>>>> Job status Succeeded >>>>>>>>>> SDK version >>>>>>>>>> Apache Beam Python 3.5 SDK 2.19.0 >>>>>>>>>> Region >>>>>>>>>> us-central1 >>>>>>>>>> Start timeFebruary 3, 2020 at 9:28:35 PM GMT-8 >>>>>>>>>> Elapsed time5 min 11 sec >>>>>>>>>> >>>>>>>>>> On Tue, Feb 4, 2020 at 9:15 AM Pablo Estrada <pabl...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Alan, >>>>>>>>>>> could it be that you're picking up the new Apache Beam 2.19.0 >>>>>>>>>>> release? Could you try depending on beam 2.18.0 to see if the issue >>>>>>>>>>> surfaces when using the new release? >>>>>>>>>>> >>>>>>>>>>> If something was working and no longer works, it sounds like a >>>>>>>>>>> bug. This may have to do with how we pickle (dill / cloudpickle) - >>>>>>>>>>> see this >>>>>>>>>>> question >>>>>>>>>>> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >>>>>>>>>>> Best >>>>>>>>>>> -P. >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 4, 2020 at 6:22 AM Alan Krumholz < >>>>>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I was running a dataflow job in GCP last night and it was >>>>>>>>>>>> running fine. >>>>>>>>>>>> This morning this same exact job is failing with the following >>>>>>>>>>>> error: >>>>>>>>>>>> >>>>>>>>>>>> Error message from worker: Traceback (most recent call last): >>>>>>>>>>>> File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>>>>>>>>>>> line 286, in loads return dill.loads(s) File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, >>>>>>>>>>>> in loads >>>>>>>>>>>> return load(file, ignore, **kwds) File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, >>>>>>>>>>>> in load >>>>>>>>>>>> return Unpickler(file, ignore=ignore, **kwds).load() File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, >>>>>>>>>>>> in load >>>>>>>>>>>> obj = StockUnpickler.load(self) File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, >>>>>>>>>>>> in >>>>>>>>>>>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' >>>>>>>>>>>> During >>>>>>>>>>>> handling of the above exception, another exception occurred: >>>>>>>>>>>> Traceback >>>>>>>>>>>> (most recent call last): File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", >>>>>>>>>>>> line 648, in do_work work_executor.execute() File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", >>>>>>>>>>>> line >>>>>>>>>>>> 176, in execute op.start() File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", >>>>>>>>>>>> line 649, in >>>>>>>>>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", line 651, in >>>>>>>>>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", line 652, in >>>>>>>>>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", line 261, in >>>>>>>>>>>> apache_beam.runners.worker.operations.Operation.start File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", line 266, in >>>>>>>>>>>> apache_beam.runners.worker.operations.Operation.start File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", line 597, in >>>>>>>>>>>> apache_beam.runners.worker.operations.DoOperation.setup File >>>>>>>>>>>> "apache_beam/runners/worker/operations.py", line 602, in >>>>>>>>>>>> apache_beam.runners.worker.operations.DoOperation.setup File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>>>>>>>>>>> line 290, in loads return dill.loads(s) File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, >>>>>>>>>>>> in loads >>>>>>>>>>>> return load(file, ignore, **kwds) File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, >>>>>>>>>>>> in load >>>>>>>>>>>> return Unpickler(file, ignore=ignore, **kwds).load() File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, >>>>>>>>>>>> in load >>>>>>>>>>>> obj = StockUnpickler.load(self) File >>>>>>>>>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, >>>>>>>>>>>> in >>>>>>>>>>>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If I use a local runner it still runs fine. >>>>>>>>>>>> Anyone else experiencing something similar today? (or know how >>>>>>>>>>>> to fix this?) >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>
