Should we update Beam's setup.py to skip this avro-python3 version? On Wed, Feb 12, 2020 at 10:57 AM Alan Krumholz <alan.krumh...@betterup.co> wrote:
> makes sense. I'll add this workaround for now. > Thanks so much for your help! > > On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> Alan, Dataflow workers preinstall Beam SDK dependencies, including (a >> working version) of avro-python3. So after reading your email once again, I >> think in your case you were not able to install Beam SDK locally. So a >> workaround for you would be to `pip install avro-python3==1.9.1` or `pip >> install pycodestyle` before installing Beam, until AVRO-2737 is resolved. >> >> >> On Wed, Feb 12, 2020 at 10:21 AM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Ah, there's already https://issues.apache.org/jira/browse/AVRO-2737 and >>> it received attention. >>> >>> On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>>> Opened https://issues.apache.org/jira/browse/AVRO-2738 >>>> >>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>>>> Here's a short repro: >>>>> >>>>> :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch >>>>> root@04b45a100d16:/# pip install avro-python3 >>>>> Collecting avro-python3 >>>>> Downloading avro-python3-1.9.2.tar.gz (37 kB) >>>>> ERROR: Command errored out with exit status 1: >>>>> command: /usr/local/bin/python -c 'import sys, setuptools, >>>>> tokenize; sys.argv[0] = >>>>> '"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"'; >>>>> __file__='"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';f=getattr(tokenize, >>>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', >>>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' >>>>> egg_info --egg-base /tmp/pip-install-mmy4vspt/avro-python3/pip-egg-info >>>>> cwd: /tmp/pip-install-mmy4vspt/avro-python3/ >>>>> Complete output (5 lines): >>>>> Traceback (most recent call last): >>>>> File "<string>", line 1, in <module> >>>>> File "/tmp/pip-install-mmy4vspt/avro-python3/setup.py", line 41, >>>>> in <module> >>>>> import pycodestyle >>>>> ModuleNotFoundError: No module named 'pycodestyle' >>>>> ---------------------------------------- >>>>> ERROR: Command errored out with exit status 1: python setup.py >>>>> egg_info Check the logs for full command output. >>>>> root@04b45a100d16:/# >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> Yes, it is a bug in the recent Avro release. We should report it >>>>>> to the Avro maintainers. The workaround is to downgrade avro-python3 to >>>>>> 1.9.1, for example via requirements.txt. >>>>>> >>>>>> On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz <sniem...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> avro-python3 1.9.2 was released on pypi 4 hours ago, and >>>>>>> added pycodestyle as a dependency, probably related? >>>>>>> >>>>>>> On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik <lc...@google.com> wrote: >>>>>>> >>>>>>>> +dev <dev@beam.apache.org> >>>>>>>> >>>>>>>> There was recently an update to add autoformatting to the Python >>>>>>>> SDK[1]. >>>>>>>> >>>>>>>> I'm seeing this during testing of a PR as well. >>>>>>>> >>>>>>>> 1: >>>>>>>> https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E >>>>>>>> >>>>>>>> On Wed, Feb 12, 2020 at 9:57 AM Alan Krumholz < >>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>> >>>>>>>>> Some more information for this as I still can't get to fix it.... >>>>>>>>> >>>>>>>>> This job is triggered using the beam[gcp] python sdk from a >>>>>>>>> KubeFlow Pipelines component which runs on top of docker image: >>>>>>>>> tensorflow/tensorflow:1.13.1-py3 >>>>>>>>> >>>>>>>>> I just checked and that image hasn't been updated recently. I also >>>>>>>>> redeployed my pipeline to another (older) deployment of KFP and it >>>>>>>>> gives me >>>>>>>>> the same error (which tells me this isn't an internal KFP problem) >>>>>>>>> >>>>>>>>> The exact same pipeline/code running on the exact same image has >>>>>>>>> been running fine for days. Did anything changed on the beam/dataflow >>>>>>>>> side >>>>>>>>> since yesterday morning? >>>>>>>>> >>>>>>>>> Thanks for your help! this is a production pipeline that is not >>>>>>>>> running for us :( >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 12, 2020 at 7:21 AM Alan Krumholz < >>>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>>> >>>>>>>>>> Hi, I have a scheduled daily job that I have been running fine in >>>>>>>>>> dataflow for days now. >>>>>>>>>> We haven't changed anything on this code but this morning run >>>>>>>>>> failed (it couldn't even spin up the job) >>>>>>>>>> The job submits a setup.py file (that also hasn't changed) but >>>>>>>>>> maybe is causing the problem? (based on the error I'm getting) >>>>>>>>>> >>>>>>>>>> Anyone else having the same issue? or know how to fix it? >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> ERROR: Complete output from command python setup.py egg_info: >>>>>>>>>> 2 ERROR: Traceback (most recent call last): >>>>>>>>>> 3 File "<string>", line 1, in <module> >>>>>>>>>> 4 File "/tmp/pip-install-42zyi89t/avro-python3/setup.py", line >>>>>>>>>> 41, in <module> >>>>>>>>>> 5 import pycodestyle >>>>>>>>>> 6 ImportError: No module named 'pycodestyle' >>>>>>>>>> 7 ---------------------------------------- >>>>>>>>>> 8ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>>> code 1 in /tmp/pip-install-42zyi89t/avro-python3/ >>>>>>>>>> 9 ERROR: Complete output from command python setup.py egg_info: >>>>>>>>>> 10 ERROR: Traceback (most recent call last): >>>>>>>>>> 11 File "<string>", line 1, in <module> >>>>>>>>>> 12 File "/tmp/pip-install-wrqytf9a/avro-python3/setup.py", line >>>>>>>>>> 41, in <module> >>>>>>>>>> 13 import pycodestyle >>>>>>>>>> 14 ImportError: No module named 'pycodestyle' >>>>>>>>>> 15 ---------------------------------------- >>>>>>>>>> 16ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>>> code 1 in /tmp/pip-install-wrqytf9a/avro-python3/ >>>>>>>>>> >>>>>>>>>
