Thats said, removing this dependency on setupVirtualEnv is definitely a good idea.
On Wed, Feb 20, 2019 at 10:14 AM Michael Luckey <[email protected]> wrote: > Is anyone using gradle for daily development of python parts? Or is this > just used as integration for Jenkins build? In case of the latter, clean > isn't called anyway, right? > > If the former, its probably worth to put more effort into python setup > anyway, e.g. declaring proper input/outputs... If done correctly (if thats > possible at all) clean will work out of the box anyway. > > On Wed, Feb 20, 2019 at 1:01 AM Udi Meiri <[email protected]> wrote: > >> I think I can solve this issue by removing the dependency and adding a >> check to see if the virtualenv was created. >> Otherwise, there shouldn't be anything to cleanup anyway. >> >> On Sat, Feb 16, 2019 at 8:04 PM Ryan Williams <[email protected]> >> wrote: >> >>> Thanks Michael, your assessment was correct. I needed python3.5 on my >>> $PATH. >>> >>> For completeness, I needed this >>> <https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib> >>> to get pyenv to install python 3.5.6 on macOS: >>> >>> ``` >>> CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6 >>> ``` >>> >>> `./gradlew clean` worked after that. >>> >>> >>> On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <[email protected]> >>> wrote: >>> >>>> As far as I understand, the build got bound to 3.5 [1]. >>>> >>>> Could it be that you do not have python3.5 on your path? e.g. try >>>> python3.5 >>>> --version >>>> >>>> If that is missing, you will not be able to run any py3 task, I guess... >>>> >>>> So, to get out of this state, you have to get python3.5 command working. >>>> >>>> At least on my machine, './gradlew clean' is working iff python3.5 is >>>> on my path. >>>> >>>> michel >>>> >>>> [1] >>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591 >>>> >>>> >>>> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <[email protected]> >>>> wrote: >>>> >>>>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on >>>>> two py3 setupVirtualEnv tasks, >>>>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and >>>>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's >>>>> full output >>>>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39> >>>>> . >>>>> >>>>> Any tips how to get out of this state, or what is causing it? >>>>> >>>>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <[email protected]> wrote: >>>>> >>>>>> Maybe add that case to >>>>>> https://issues.apache.org/jira/browse/BEAM-6459. >>>>>> >>>>>> Kenn >>>>>> >>>>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <[email protected]> wrote: >>>>>> >>>>>>> Probably related, a top level ./gradlew clean fails with the >>>>>>> following: >>>>>>> >>>>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED >>>>>>> The path python3.5 (from --python=python3.5) does not exist >>>>>>> >>>>>>> Can we just limit clean to do cleanup?! >>>>>>> >>>>>>> >>>>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Just some background, grpcio-tools is what's used to do the proto >>>>>>>> generation. Unfortunately it's expensive to compile and doesn't >>>>>>>> provide very many wheels, so we want to install it once, not every >>>>>>>> time. (It's also used in more than just tests; one needs it every >>>>>>>> time >>>>>>>> the .proto files change.) >>>>>>>> >>>>>>>> That being said, we could probably do a much cheaper clean. >>>>>>>> >>>>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <[email protected]> wrote: >>>>>>>> > >>>>>>>> > grpcio-tools could probably be moved under the "test" tag in >>>>>>>> setup.py. Not sure why it has to be specified in gradle configs. >>>>>>>> > >>>>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <[email protected]> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean` >>>>>>>> without installing gcpio-tools, etc? >>>>>>>> >> >>>>>>>> >> Kenn >>>>>>>> >> >>>>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <[email protected]> >>>>>>>> wrote: >>>>>>>> >>> >>>>>>>> >>> setup.py has requirements like setuptools, which are installed >>>>>>>> in the virtual environment. >>>>>>>> >>> So even running the clean command requires the virtualenv to be >>>>>>>> set up. >>>>>>>> >>> >>>>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython >>>>>>>> if setupVirtualenv has not been run. (perhaps by checking for the >>>>>>>> existence >>>>>>>> of its output directory) >>>>>>>> >>> >>>>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to >>>>>>>> record the conclusion. Doesn't require Beam knowledge so I labeled >>>>>>>> "starter". >>>>>>>> >>>> >>>>>>>> >>>> Kenn >>>>>>>> >>>> >>>>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> This seems to be on purpose [1] >>>>>>>> >>>>> >>>>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. >>>>>>>> We probably should work around that. >>>>>>>> >>>>> >>>>>>>> >>>>> [1] >>>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610 >>>>>>>> >>>>> >>>>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean` >>>>>>>> fails due to failure of `setupVirtualEnv` tasks. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Manu Zhang >>>>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles < >>>>>>>> [email protected]>, wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` >>>>>>>> tasks that invoke things such as `setup.py bdist_wheel for >>>>>>>> grpcio-tools`. >>>>>>>> Overall it took 4 minutes. Is this intended? >>>>>>>> >>>>>> >>>>>>>> >>>>>> Kenn >>>>>>>> >>>>>>>
