Chad, I believe the answer is the "setup_requires" line is causing the sdks/python/.eggs directory to be created.
This command fails with the setup_requires line (same Errno 17), but succeeds without it: $ \rm -r .eggs/; ../../gradlew installGcpTest [~8 failed tasks] $ ls .eggs pytest_runner-5.2-py2.7.egg pytest_runner-5.2-py3.5.egg pytest_runner-5.2-py3.6.egg pytest_runner-5.2-py3.7.egg README.txt I'll go ahead and create a PR to remove setup_requires from setup.py. On Tue, Nov 26, 2019 at 4:16 PM Chad Dombrova <chad...@gmail.com> wrote: > It seems like the offending packages are those that only have source > distributions (i.e. no wheels). But why are the eggs being installed in > sdks/python/.eggs instead of into the virtualenv created by setupVirtualenv > gradle task or by tox? > > > On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri <eh...@google.com> wrote: > >> Basically, I believe what's happening is that a new Gradle task was added >> that uses setup.py but doesn't have the same dependency on some main >> setup.py task that all others depend on (list sdist). >> >> On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri <eh...@google.com> wrote: >> >>> Correction: the error is not gone after removing the line. I get instead: >>> error: [Errno 17] File exists: >>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg' >>> >>> >>> On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri <eh...@google.com> wrote: >>> >>>> I managed to recreate one of the issues with this command: >>>> ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do echo >>>> "python setup.py -q nosetests --tests >>>> apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh ; done >>>> >>>> This reliably gives me: >>>> OSError: [Errno 17] File exists: >>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/pytest_runner-5.2-py2.7.egg' >>>> >>>> If I remove this line from setup.py the error is gone: >>>> setup_requires=['pytest_runner'], >>>> >>>> >>>> On Tue, Nov 26, 2019 at 2:54 PM Chad Dombrova <chad...@gmail.com> >>>> wrote: >>>> >>>>> Thanks for looking into this. It seems like it might be something to >>>>> do with data that is cached on the Jenkins slaves between runs, which may >>>>> be what prevents this from showing up locally? >>>>> >>>>> If your theory about setuptools is correct, and it sounds likely, we >>>>> should be able to lock down the version, which we should definitely be >>>>> doing for all of our dependencies. >>>>> >>>>> -chad >>>>> >>>>> >>>>> >>>>> On Tue, Nov 26, 2019 at 1:33 PM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> I tried to debug but did not make much progress. I cannot reproduce >>>>>> locally, however all python precommits and postcommits are failing. >>>>>> >>>>>> One guess is, setuptools released a new version that does not support >>>>>> eggs a few days ago, that might be the cause ( >>>>>> https://github.com/pypa/setuptools/blob/master/CHANGES.rst) but that >>>>>> should have reproduced locally. >>>>>> Maybe something is wrong with the jenkins machines, and we could >>>>>> perhaps bring them to a clean state. >>>>>> >>>>>> I suspected this being related to pytest somehow (as the first 4 >>>>>> JIRAs had pytest in the error line) but the error Chad saw is different. >>>>>> >>>>>> +Valentyn Tymofieiev <valen...@google.com> and +Yifan Zou >>>>>> <yifan...@google.com> could you help with looking into this? >>>>>> >>>>>> >>>>>> Ahmet >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Nov 26, 2019 at 9:14 AM Luke Cwik <lc...@google.com> wrote: >>>>>> >>>>>>> I also started to see this on PRs that I'm reviewing. >>>>>>> BEAM-8793, BEAM-8653, BEAM-8631, BEAM-8249 mention issues with setup.py >>>>>>> and >>>>>>> egg_info but this looks different then all of those so I filed >>>>>>> BEAM-8831. >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 25, 2019 at 10:27 PM Chad Dombrova <chad...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Actually, it looks like I'm getting the same error on multiple PRs: >>>>>>>> https://scans.gradle.com/s/ihfmrxr7evslw >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Nov 25, 2019 at 10:26 PM Chad Dombrova <chad...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> The cython tests started failing on one of my PRs which were >>>>>>>>> succeeding before. The error is one that I've never seen before >>>>>>>>> (separated onto different lines to make it easier to read): >>>>>>>>> >>>>>>>>> Caused by: org.gradle.api.GradleException: >>>>>>>>> Could not copy file >>>>>>>>> >>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2 >>>>>>>>> /src/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg' >>>>>>>>> to >>>>>>>>> >>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2 >>>>>>>>> /src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg'. >>>>>>>>> >>>>>>>>> Followed immediately by an error about could not create a >>>>>>>>> directory of the same name. Here's the gradle scan: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://scans.gradle.com/s/ihfmrxr7evslw/failure?openFailures=WzFd&openStackTraces=WzZd#top=0 >>>>>>>>> >>>>>>>>> Any ideas? >>>>>>>>> >>>>>>>>> -chad >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
