I'm not sure about where the error with the simplegeneric, timeloop .eggs directories come from, but I did figure out that they don't get installed as eggs if you add them to the "test" extras in setup.py, e.g.:
extras_require={
'docs': ['Sphinx>=1.5.2,<2.0'],
'test': REQUIRED_TEST_PACKAGES + INTERACTIVE_BEAM,
'gcp': GCP_REQUIREMENTS,
'interactive': INTERACTIVE_BEAM,
},
This is further proof of the wisdom of the pytest-runner deprecation notice
<https://pypi.org/project/pytest-runner/> (emphasis mine):
"""
Remove ‘pytest’ and any other testing requirements from ‘*tests_require*’,
preferably removing the setup_requires option.
"""
I believe we don't rely on the tests_require definition. Removing it might
break developers running "python setup.py test", but the alternative is a
simple "setup.py && pip install".
On Tue, Nov 26, 2019 at 5:14 PM Chad Dombrova <chad...@gmail.com> wrote:
> Sorry wrong link: https://github.com/apache/beam/pull/9915
>
>
>
> On Tue, Nov 26, 2019 at 5:12 PM Udi Meiri <eh...@google.com> wrote:
>
>> I looked at #9959 but it doesn't seem to modify setup.py?
>> The additional eggs for timeloop etc. are troubling though. Not sure
>> where those come from.
>>
>> On Tue, Nov 26, 2019 at 4:59 PM Chad Dombrova <chad...@gmail.com> wrote:
>>
>>> Is setup_requires being used somewhere else, because I'm still getting
>>> errors after removing it from sdks/python/setup.py.
>>>
>>> I removed it from this PR: https://github.com/apache/beam/pull/9959
>>>
>>> Here's the gradle scan:
>>> https://scans.gradle.com/s/oinh5xpaly3dk/failure#top=0
>>>
>>> The error shows up differently than before when
>>> setup_requries=['pytest-runner'] was present -- it's in a gradle traceback
>>> now rather than the console log. I've also seen different packages listed
>>> as the culprit (simplegeneric, timeloop).
>>>
>>> -chad
>>>
>>>
>>>
>>> On Tue, Nov 26, 2019 at 4:47 PM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> Chad, I believe the answer is the "setup_requires" line is causing the
>>>> sdks/python/.eggs directory to be created.
>>>>
>>>> This command fails with the setup_requires line (same Errno 17), but
>>>> succeeds without it:
>>>> $ \rm -r .eggs/; ../../gradlew installGcpTest
>>>> [~8 failed tasks]
>>>> $ ls .eggs
>>>> pytest_runner-5.2-py2.7.egg pytest_runner-5.2-py3.5.egg
>>>> pytest_runner-5.2-py3.6.egg pytest_runner-5.2-py3.7.egg README.txt
>>>>
>>>> I'll go ahead and create a PR to remove setup_requires from setup.py.
>>>>
>>>> On Tue, Nov 26, 2019 at 4:16 PM Chad Dombrova <chad...@gmail.com>
>>>> wrote:
>>>>
>>>>> It seems like the offending packages are those that only have source
>>>>> distributions (i.e. no wheels). But why are the eggs being installed in
>>>>> sdks/python/.eggs instead of into the virtualenv created by
>>>>> setupVirtualenv
>>>>> gradle task or by tox?
>>>>>
>>>>>
>>>>> On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> Basically, I believe what's happening is that a new Gradle task was
>>>>>> added that uses setup.py but doesn't have the same dependency on some
>>>>>> main
>>>>>> setup.py task that all others depend on (list sdist).
>>>>>>
>>>>>> On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>
>>>>>>> Correction: the error is not gone after removing the line. I get
>>>>>>> instead:
>>>>>>> error: [Errno 17] File exists:
>>>>>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg'
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>>
>>>>>>>> I managed to recreate one of the issues with this command:
>>>>>>>> ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do echo
>>>>>>>> "python setup.py -q nosetests --tests
>>>>>>>> apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh ;
>>>>>>>> done
>>>>>>>>
>>>>>>>> This reliably gives me:
>>>>>>>> OSError: [Errno 17] File exists:
>>>>>>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/pytest_runner-5.2-py2.7.egg'
>>>>>>>>
>>>>>>>> If I remove this line from setup.py the error is gone:
>>>>>>>> setup_requires=['pytest_runner'],
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 26, 2019 at 2:54 PM Chad Dombrova <chad...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for looking into this. It seems like it might be something
>>>>>>>>> to do with data that is cached on the Jenkins slaves between runs,
>>>>>>>>> which
>>>>>>>>> may be what prevents this from showing up locally?
>>>>>>>>>
>>>>>>>>> If your theory about setuptools is correct, and it sounds likely,
>>>>>>>>> we should be able to lock down the version, which we should
>>>>>>>>> definitely be
>>>>>>>>> doing for all of our dependencies.
>>>>>>>>>
>>>>>>>>> -chad
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Nov 26, 2019 at 1:33 PM Ahmet Altay <al...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I tried to debug but did not make much progress. I cannot
>>>>>>>>>> reproduce locally, however all python precommits and postcommits are
>>>>>>>>>> failing.
>>>>>>>>>>
>>>>>>>>>> One guess is, setuptools released a new version that does not
>>>>>>>>>> support eggs a few days ago, that might be the cause (
>>>>>>>>>> https://github.com/pypa/setuptools/blob/master/CHANGES.rst) but
>>>>>>>>>> that should have reproduced locally.
>>>>>>>>>> Maybe something is wrong with the jenkins machines, and we could
>>>>>>>>>> perhaps bring them to a clean state.
>>>>>>>>>>
>>>>>>>>>> I suspected this being related to pytest somehow (as the first 4
>>>>>>>>>> JIRAs had pytest in the error line) but the error Chad saw is
>>>>>>>>>> different.
>>>>>>>>>>
>>>>>>>>>> +Valentyn Tymofieiev <valen...@google.com> and +Yifan Zou
>>>>>>>>>> <yifan...@google.com> could you help with looking into this?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ahmet
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 26, 2019 at 9:14 AM Luke Cwik <lc...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I also started to see this on PRs that I'm reviewing.
>>>>>>>>>>> BEAM-8793, BEAM-8653, BEAM-8631, BEAM-8249 mention issues with
>>>>>>>>>>> setup.py and
>>>>>>>>>>> egg_info but this looks different then all of those so I filed
>>>>>>>>>>> BEAM-8831.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 25, 2019 at 10:27 PM Chad Dombrova <
>>>>>>>>>>> chad...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Actually, it looks like I'm getting the same error on multiple
>>>>>>>>>>>> PRs: https://scans.gradle.com/s/ihfmrxr7evslw
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 25, 2019 at 10:26 PM Chad Dombrova <
>>>>>>>>>>>> chad...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>> The cython tests started failing on one of my PRs which were
>>>>>>>>>>>>> succeeding before. The error is one that I've never seen before
>>>>>>>>>>>>> (separated onto different lines to make it easier to read):
>>>>>>>>>>>>>
>>>>>>>>>>>>> Caused by: org.gradle.api.GradleException:
>>>>>>>>>>>>> Could not copy file
>>>>>>>>>>>>>
>>>>>>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2
>>>>>>>>>>>>> /src/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg'
>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2
>>>>>>>>>>>>> /src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg'.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Followed immediately by an error about could not create a
>>>>>>>>>>>>> directory of the same name. Here's the gradle scan:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://scans.gradle.com/s/ihfmrxr7evslw/failure?openFailures=WzFd&openStackTraces=WzZd#top=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any ideas?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -chad
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
