Alan, Dataflow workers preinstall Beam SDK dependencies, including (a working version) of avro-python3. So after reading your email once again, I think in your case you were not able to install Beam SDK locally. So a workaround for you would be to `pip install avro-python3==1.9.1` or `pip install pycodestyle` before installing Beam, until AVRO-2737 is resolved.
On Wed, Feb 12, 2020 at 10:21 AM Valentyn Tymofieiev <valen...@google.com> wrote: > Ah, there's already https://issues.apache.org/jira/browse/AVRO-2737 and > it received attention. > > On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> Opened https://issues.apache.org/jira/browse/AVRO-2738 >> >> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Here's a short repro: >>> >>> :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch >>> root@04b45a100d16:/# pip install avro-python3 >>> Collecting avro-python3 >>> Downloading avro-python3-1.9.2.tar.gz (37 kB) >>> ERROR: Command errored out with exit status 1: >>> command: /usr/local/bin/python -c 'import sys, setuptools, >>> tokenize; sys.argv[0] = >>> '"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"'; >>> __file__='"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';f=getattr(tokenize, >>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', >>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' >>> egg_info --egg-base /tmp/pip-install-mmy4vspt/avro-python3/pip-egg-info >>> cwd: /tmp/pip-install-mmy4vspt/avro-python3/ >>> Complete output (5 lines): >>> Traceback (most recent call last): >>> File "<string>", line 1, in <module> >>> File "/tmp/pip-install-mmy4vspt/avro-python3/setup.py", line 41, >>> in <module> >>> import pycodestyle >>> ModuleNotFoundError: No module named 'pycodestyle' >>> ---------------------------------------- >>> ERROR: Command errored out with exit status 1: python setup.py egg_info >>> Check the logs for full command output. >>> root@04b45a100d16:/# >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>>> Yes, it is a bug in the recent Avro release. We should report it to the >>>> Avro maintainers. The workaround is to downgrade avro-python3 to 1.9.1, for >>>> example via requirements.txt. >>>> >>>> On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz <sniem...@apache.org> >>>> wrote: >>>> >>>>> avro-python3 1.9.2 was released on pypi 4 hours ago, and >>>>> added pycodestyle as a dependency, probably related? >>>>> >>>>> On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik <lc...@google.com> wrote: >>>>> >>>>>> +dev <d...@beam.apache.org> >>>>>> >>>>>> There was recently an update to add autoformatting to the Python >>>>>> SDK[1]. >>>>>> >>>>>> I'm seeing this during testing of a PR as well. >>>>>> >>>>>> 1: >>>>>> https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E >>>>>> >>>>>> On Wed, Feb 12, 2020 at 9:57 AM Alan Krumholz < >>>>>> alan.krumh...@betterup.co> wrote: >>>>>> >>>>>>> Some more information for this as I still can't get to fix it.... >>>>>>> >>>>>>> This job is triggered using the beam[gcp] python sdk from a KubeFlow >>>>>>> Pipelines component which runs on top of docker image: >>>>>>> tensorflow/tensorflow:1.13.1-py3 >>>>>>> >>>>>>> I just checked and that image hasn't been updated recently. I also >>>>>>> redeployed my pipeline to another (older) deployment of KFP and it >>>>>>> gives me >>>>>>> the same error (which tells me this isn't an internal KFP problem) >>>>>>> >>>>>>> The exact same pipeline/code running on the exact same image has >>>>>>> been running fine for days. Did anything changed on the beam/dataflow >>>>>>> side >>>>>>> since yesterday morning? >>>>>>> >>>>>>> Thanks for your help! this is a production pipeline that is not >>>>>>> running for us :( >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 12, 2020 at 7:21 AM Alan Krumholz < >>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>> >>>>>>>> Hi, I have a scheduled daily job that I have been running fine in >>>>>>>> dataflow for days now. >>>>>>>> We haven't changed anything on this code but this morning run >>>>>>>> failed (it couldn't even spin up the job) >>>>>>>> The job submits a setup.py file (that also hasn't changed) but >>>>>>>> maybe is causing the problem? (based on the error I'm getting) >>>>>>>> >>>>>>>> Anyone else having the same issue? or know how to fix it? >>>>>>>> Thanks! >>>>>>>> >>>>>>>> ERROR: Complete output from command python setup.py egg_info: >>>>>>>> 2 ERROR: Traceback (most recent call last): >>>>>>>> 3 File "<string>", line 1, in <module> >>>>>>>> 4 File "/tmp/pip-install-42zyi89t/avro-python3/setup.py", line 41, >>>>>>>> in <module> >>>>>>>> 5 import pycodestyle >>>>>>>> 6 ImportError: No module named 'pycodestyle' >>>>>>>> 7 ---------------------------------------- >>>>>>>> 8ERROR: Command "python setup.py egg_info" failed with error code >>>>>>>> 1 in /tmp/pip-install-42zyi89t/avro-python3/ >>>>>>>> 9 ERROR: Complete output from command python setup.py egg_info: >>>>>>>> 10 ERROR: Traceback (most recent call last): >>>>>>>> 11 File "<string>", line 1, in <module> >>>>>>>> 12 File "/tmp/pip-install-wrqytf9a/avro-python3/setup.py", line >>>>>>>> 41, in <module> >>>>>>>> 13 import pycodestyle >>>>>>>> 14 ImportError: No module named 'pycodestyle' >>>>>>>> 15 ---------------------------------------- >>>>>>>> 16ERROR: Command "python setup.py egg_info" failed with error code >>>>>>>> 1 in /tmp/pip-install-wrqytf9a/avro-python3/ >>>>>>>> >>>>>>>
