yeah, I've excised both test_requires and setup_requires in my test
simplification PR: https://github.com/apache/beam/pull/10038
I'm happy to see those go sooner rather than later, as it'll reduce the
scope of my PR. The rest of my PR is about ensuring that build
dependencies like cython and grpc are available at "build" time (i.e. when
setup.py gets called), and the modern solution for this is a
pep517/518-compliant build system, of which tox is one.
-chad
On Tue, Nov 26, 2019 at 6:39 PM Udi Meiri <eh...@google.com> wrote:
> I'm not sure about where the error with the simplegeneric, timeloop .eggs
> directories come from,
> but I did figure out that they don't get installed as eggs if you add them
> to the "test" extras in setup.py, e.g.:
>
> extras_require={
> 'docs': ['Sphinx>=1.5.2,<2.0'],
> 'test': REQUIRED_TEST_PACKAGES + INTERACTIVE_BEAM,
> 'gcp': GCP_REQUIREMENTS,
> 'interactive': INTERACTIVE_BEAM,
> },
>
> This is further proof of the wisdom of the pytest-runner deprecation
> notice <https://pypi.org/project/pytest-runner/> (emphasis mine):
> """
> Remove ‘pytest’ and any other testing requirements from ‘*tests_require*’,
> preferably removing the setup_requires option.
> """
>
> I believe we don't rely on the tests_require definition. Removing it might
> break developers running "python setup.py test", but the alternative is a
> simple "setup.py && pip install".
>
>
> On Tue, Nov 26, 2019 at 5:14 PM Chad Dombrova <chad...@gmail.com> wrote:
>
>> Sorry wrong link: https://github.com/apache/beam/pull/9915
>>
>>
>>
>> On Tue, Nov 26, 2019 at 5:12 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> I looked at #9959 but it doesn't seem to modify setup.py?
>>> The additional eggs for timeloop etc. are troubling though. Not sure
>>> where those come from.
>>>
>>> On Tue, Nov 26, 2019 at 4:59 PM Chad Dombrova <chad...@gmail.com> wrote:
>>>
>>>> Is setup_requires being used somewhere else, because I'm still getting
>>>> errors after removing it from sdks/python/setup.py.
>>>>
>>>> I removed it from this PR: https://github.com/apache/beam/pull/9959
>>>>
>>>> Here's the gradle scan:
>>>> https://scans.gradle.com/s/oinh5xpaly3dk/failure#top=0
>>>>
>>>> The error shows up differently than before when
>>>> setup_requries=['pytest-runner'] was present -- it's in a gradle traceback
>>>> now rather than the console log. I've also seen different packages listed
>>>> as the culprit (simplegeneric, timeloop).
>>>>
>>>> -chad
>>>>
>>>>
>>>>
>>>> On Tue, Nov 26, 2019 at 4:47 PM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> Chad, I believe the answer is the "setup_requires" line is causing the
>>>>> sdks/python/.eggs directory to be created.
>>>>>
>>>>> This command fails with the setup_requires line (same Errno 17), but
>>>>> succeeds without it:
>>>>> $ \rm -r .eggs/; ../../gradlew installGcpTest
>>>>> [~8 failed tasks]
>>>>> $ ls .eggs
>>>>> pytest_runner-5.2-py2.7.egg pytest_runner-5.2-py3.5.egg
>>>>> pytest_runner-5.2-py3.6.egg pytest_runner-5.2-py3.7.egg README.txt
>>>>>
>>>>> I'll go ahead and create a PR to remove setup_requires from setup.py.
>>>>>
>>>>> On Tue, Nov 26, 2019 at 4:16 PM Chad Dombrova <chad...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> It seems like the offending packages are those that only have source
>>>>>> distributions (i.e. no wheels). But why are the eggs being installed in
>>>>>> sdks/python/.eggs instead of into the virtualenv created by
>>>>>> setupVirtualenv
>>>>>> gradle task or by tox?
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>
>>>>>>> Basically, I believe what's happening is that a new Gradle task was
>>>>>>> added that uses setup.py but doesn't have the same dependency on some
>>>>>>> main
>>>>>>> setup.py task that all others depend on (list sdist).
>>>>>>>
>>>>>>> On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>>
>>>>>>>> Correction: the error is not gone after removing the line. I get
>>>>>>>> instead:
>>>>>>>> error: [Errno 17] File exists:
>>>>>>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg'
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>>>
>>>>>>>>> I managed to recreate one of the issues with this command:
>>>>>>>>> ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do
>>>>>>>>> echo "python setup.py -q nosetests --tests
>>>>>>>>> apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh
>>>>>>>>> ; done
>>>>>>>>>
>>>>>>>>> This reliably gives me:
>>>>>>>>> OSError: [Errno 17] File exists:
>>>>>>>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/pytest_runner-5.2-py2.7.egg'
>>>>>>>>>
>>>>>>>>> If I remove this line from setup.py the error is gone:
>>>>>>>>> setup_requires=['pytest_runner'],
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Nov 26, 2019 at 2:54 PM Chad Dombrova <chad...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for looking into this. It seems like it might be something
>>>>>>>>>> to do with data that is cached on the Jenkins slaves between runs,
>>>>>>>>>> which
>>>>>>>>>> may be what prevents this from showing up locally?
>>>>>>>>>>
>>>>>>>>>> If your theory about setuptools is correct, and it sounds likely,
>>>>>>>>>> we should be able to lock down the version, which we should
>>>>>>>>>> definitely be
>>>>>>>>>> doing for all of our dependencies.
>>>>>>>>>>
>>>>>>>>>> -chad
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 26, 2019 at 1:33 PM Ahmet Altay <al...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I tried to debug but did not make much progress. I cannot
>>>>>>>>>>> reproduce locally, however all python precommits and postcommits are
>>>>>>>>>>> failing.
>>>>>>>>>>>
>>>>>>>>>>> One guess is, setuptools released a new version that does not
>>>>>>>>>>> support eggs a few days ago, that might be the cause (
>>>>>>>>>>> https://github.com/pypa/setuptools/blob/master/CHANGES.rst) but
>>>>>>>>>>> that should have reproduced locally.
>>>>>>>>>>> Maybe something is wrong with the jenkins machines, and we could
>>>>>>>>>>> perhaps bring them to a clean state.
>>>>>>>>>>>
>>>>>>>>>>> I suspected this being related to pytest somehow (as the first 4
>>>>>>>>>>> JIRAs had pytest in the error line) but the error Chad saw is
>>>>>>>>>>> different.
>>>>>>>>>>>
>>>>>>>>>>> +Valentyn Tymofieiev <valen...@google.com> and +Yifan Zou
>>>>>>>>>>> <yifan...@google.com> could you help with looking into this?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ahmet
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Nov 26, 2019 at 9:14 AM Luke Cwik <lc...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I also started to see this on PRs that I'm reviewing.
>>>>>>>>>>>> BEAM-8793, BEAM-8653, BEAM-8631, BEAM-8249 mention issues with
>>>>>>>>>>>> setup.py and
>>>>>>>>>>>> egg_info but this looks different then all of those so I filed
>>>>>>>>>>>> BEAM-8831.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 25, 2019 at 10:27 PM Chad Dombrova <
>>>>>>>>>>>> chad...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Actually, it looks like I'm getting the same error on multiple
>>>>>>>>>>>>> PRs: https://scans.gradle.com/s/ihfmrxr7evslw
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Nov 25, 2019 at 10:26 PM Chad Dombrova <
>>>>>>>>>>>>> chad...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>> The cython tests started failing on one of my PRs which were
>>>>>>>>>>>>>> succeeding before. The error is one that I've never seen before
>>>>>>>>>>>>>> (separated onto different lines to make it easier to read):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Caused by: org.gradle.api.GradleException:
>>>>>>>>>>>>>> Could not copy file
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2
>>>>>>>>>>>>>> /src/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg'
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2
>>>>>>>>>>>>>> /src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg'.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Followed immediately by an error about could not create a
>>>>>>>>>>>>>> directory of the same name. Here's the gradle scan:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://scans.gradle.com/s/ihfmrxr7evslw/failure?openFailures=WzFd&openStackTraces=WzZd#top=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any ideas?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -chad
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
