makes sense. I'll add this workaround for now. Thanks so much for your help!
On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev <valen...@google.com> wrote: > Alan, Dataflow workers preinstall Beam SDK dependencies, including (a > working version) of avro-python3. So after reading your email once again, I > think in your case you were not able to install Beam SDK locally. So a > workaround for you would be to `pip install avro-python3==1.9.1` or `pip > install pycodestyle` before installing Beam, until AVRO-2737 is resolved. > > > On Wed, Feb 12, 2020 at 10:21 AM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> Ah, there's already https://issues.apache.org/jira/browse/AVRO-2737 and >> it received attention. >> >> On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Opened https://issues.apache.org/jira/browse/AVRO-2738 >>> >>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>>> Here's a short repro: >>>> >>>> :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch >>>> root@04b45a100d16:/# pip install avro-python3 >>>> Collecting avro-python3 >>>> Downloading avro-python3-1.9.2.tar.gz (37 kB) >>>> ERROR: Command errored out with exit status 1: >>>> command: /usr/local/bin/python -c 'import sys, setuptools, >>>> tokenize; sys.argv[0] = >>>> '"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"'; >>>> __file__='"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';f=getattr(tokenize, >>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', >>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' >>>> egg_info --egg-base /tmp/pip-install-mmy4vspt/avro-python3/pip-egg-info >>>> cwd: /tmp/pip-install-mmy4vspt/avro-python3/ >>>> Complete output (5 lines): >>>> Traceback (most recent call last): >>>> File "<string>", line 1, in <module> >>>> File "/tmp/pip-install-mmy4vspt/avro-python3/setup.py", line 41, >>>> in <module> >>>> import pycodestyle >>>> ModuleNotFoundError: No module named 'pycodestyle' >>>> ---------------------------------------- >>>> ERROR: Command errored out with exit status 1: python setup.py egg_info >>>> Check the logs for full command output. >>>> root@04b45a100d16:/# >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>>>> Yes, it is a bug in the recent Avro release. We should report it >>>>> to the Avro maintainers. The workaround is to downgrade avro-python3 to >>>>> 1.9.1, for example via requirements.txt. >>>>> >>>>> On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz <sniem...@apache.org> >>>>> wrote: >>>>> >>>>>> avro-python3 1.9.2 was released on pypi 4 hours ago, and >>>>>> added pycodestyle as a dependency, probably related? >>>>>> >>>>>> On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik <lc...@google.com> wrote: >>>>>> >>>>>>> +dev <d...@beam.apache.org> >>>>>>> >>>>>>> There was recently an update to add autoformatting to the Python >>>>>>> SDK[1]. >>>>>>> >>>>>>> I'm seeing this during testing of a PR as well. >>>>>>> >>>>>>> 1: >>>>>>> https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E >>>>>>> >>>>>>> On Wed, Feb 12, 2020 at 9:57 AM Alan Krumholz < >>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>> >>>>>>>> Some more information for this as I still can't get to fix it.... >>>>>>>> >>>>>>>> This job is triggered using the beam[gcp] python sdk from a >>>>>>>> KubeFlow Pipelines component which runs on top of docker image: >>>>>>>> tensorflow/tensorflow:1.13.1-py3 >>>>>>>> >>>>>>>> I just checked and that image hasn't been updated recently. I also >>>>>>>> redeployed my pipeline to another (older) deployment of KFP and it >>>>>>>> gives me >>>>>>>> the same error (which tells me this isn't an internal KFP problem) >>>>>>>> >>>>>>>> The exact same pipeline/code running on the exact same image has >>>>>>>> been running fine for days. Did anything changed on the beam/dataflow >>>>>>>> side >>>>>>>> since yesterday morning? >>>>>>>> >>>>>>>> Thanks for your help! this is a production pipeline that is not >>>>>>>> running for us :( >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 12, 2020 at 7:21 AM Alan Krumholz < >>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>> >>>>>>>>> Hi, I have a scheduled daily job that I have been running fine in >>>>>>>>> dataflow for days now. >>>>>>>>> We haven't changed anything on this code but this morning run >>>>>>>>> failed (it couldn't even spin up the job) >>>>>>>>> The job submits a setup.py file (that also hasn't changed) but >>>>>>>>> maybe is causing the problem? (based on the error I'm getting) >>>>>>>>> >>>>>>>>> Anyone else having the same issue? or know how to fix it? >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> ERROR: Complete output from command python setup.py egg_info: >>>>>>>>> 2 ERROR: Traceback (most recent call last): >>>>>>>>> 3 File "<string>", line 1, in <module> >>>>>>>>> 4 File "/tmp/pip-install-42zyi89t/avro-python3/setup.py", line >>>>>>>>> 41, in <module> >>>>>>>>> 5 import pycodestyle >>>>>>>>> 6 ImportError: No module named 'pycodestyle' >>>>>>>>> 7 ---------------------------------------- >>>>>>>>> 8ERROR: Command "python setup.py egg_info" failed with error code >>>>>>>>> 1 in /tmp/pip-install-42zyi89t/avro-python3/ >>>>>>>>> 9 ERROR: Complete output from command python setup.py egg_info: >>>>>>>>> 10 ERROR: Traceback (most recent call last): >>>>>>>>> 11 File "<string>", line 1, in <module> >>>>>>>>> 12 File "/tmp/pip-install-wrqytf9a/avro-python3/setup.py", line >>>>>>>>> 41, in <module> >>>>>>>>> 13 import pycodestyle >>>>>>>>> 14 ImportError: No module named 'pycodestyle' >>>>>>>>> 15 ---------------------------------------- >>>>>>>>> 16ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>> code 1 in /tmp/pip-install-wrqytf9a/avro-python3/ >>>>>>>>> >>>>>>>>
