I could not find a way for the hidden pip install to succeed within the virtual environment.
Also, heads up that we found another issue related to timer encoding that looks like a release blocker. Details to follow. On Fri, May 8, 2020 at 9:53 AM Udi Meiri <[email protected]> wrote: > +Chad Dombrova <[email protected]> , who added _find_protoc_gen_mypy. > > I'm guessing that the code > in _install_grpcio_tools_and_generate_proto_files creates a kind of > virtualenv, but it only works well for staging Python modules and not > binaries like protoc-gen-mypy. > (I assume there's a reason why it doesn't invoke virtualenv, probably > since the list of things setup.py can expect to be installed is very > minimal (setuptools).) > > One solution would be to make these setup.py dependencies explicit in > pyproject.toml, such that pip installs them before running setup.py: > https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support > It would help when using tools like pip ("pip wheel"), but I'm not sure > what the alternative for "python setup.py sdist" is. > > > On Thu, May 7, 2020 at 10:40 PM Thomas Weise <[email protected]> wrote: > >> No additional stacktraces. Full error output below. >> >> It's not clear what is going wrong. >> >> There isn't any exception from the subprocess execution since the >> "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed. >> >> Also, the time it takes to perform the install is equivalent to >> successfully running the pip command. >> >> I will report back if I find anything else. Currently doing the >> explicit install via pip install -r sdks/python/build-requirements.txt >> >> Thanks, >> Thomas >> >> WARNING:root:Installing grpcio-tools took 269.27 seconds. >> INFO:gen_protos:Regenerating Python proto definitions (no output files). >> Process Process-1: >> Traceback (most recent call last): >> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in >> _bootstrap >> self.run() >> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run >> self._target(*self._args, **self._kwargs) >> File >> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >> 378, in _install_grpcio_tools_and_generate_proto_files >> generate_proto_files(force=force) >> File >> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >> 315, in generate_proto_files >> protoc_gen_mypy = _find_protoc_gen_mypy() >> File >> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >> 233, in _find_protoc_gen_mypy >> (fname, ', '.join(search_paths))) >> RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin, >> /code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin, >> /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin >> Traceback (most recent call last): >> File "setup.py", line 311, in <module> >> 'mypy': generate_protos_first(mypy), >> File >> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py", >> line 129, in setup >> return distutils.core.setup(**attrs) >> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup >> dist.run_commands() >> File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands >> self.run_command(cmd) >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command >> cmd_obj.run() >> File >> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", >> line 204, in run >> self.run_command('build') >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command >> self.distribution.run_command(command) >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command >> cmd_obj.run() >> File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run >> self.run_command(cmd_name) >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command >> self.distribution.run_command(command) >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command >> cmd_obj.run() >> File "setup.py", line 235, in run >> gen_protos.generate_proto_files() >> File >> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >> 310, in generate_proto_files >> raise ValueError("Proto generation failed (see log for details).") >> ValueError: Proto generation failed (see log for details). >> >> >> On Thu, May 7, 2020 at 2:25 PM Udi Meiri <[email protected]> wrote: >> >>> It's hard to say without more details what's going on. Ahmet you're >>> right that it installs build-requirements.txt and retries calling >>> generate_proto_files(). >>> >>> Thomas, were there additional stacktraces? (after a "During handling of >>> the above exception, another exception occurred:" message?) >>> >>> >>> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <[email protected]> wrote: >>> >>>> >>>> >>>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <[email protected]> wrote: >>>> >>>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where >>>>> build-requirements.txt didn't exist. >>>>> >>>>> Is there a reason why this cannot happen automatically when >>>>> running python3.6 setup.py sdist bdist_wheel ? >>>>> >>>> >>>> I _believe_ this should happen automatically here: >>>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365. >>>> Maybe there is a problem there? >>>> >>>> >>>>> >>>>> Thomas >>>>> >>>>> >>>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <[email protected]> wrote: >>>>> >>>>>> Probably not the issue, but double checking: are you running "pip >>>>>> install -r sdks/python/build-requirements.txt" first? >>>>>> >>>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <[email protected]> wrote: >>>>>> >>>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem >>>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No >>>>>>> module named 'grpc_tools' *(see details below) >>>>>>> >>>>>>> I cannot reproduce this locally. >>>>>>> >>>>>>> Any =suggestions on what to look for? >>>>>>> >>>>>>> Thanks, >>>>>>> Thomas >>>>>>> >>>>>>> [?25hBuilding wheels for collected packages: future, gr >>>>>>> Running setup.py bdist_wheel for future ... [?25l- \ >>>>>>> [?25h Stored in directory: >>>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1 >>>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>> / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>> / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>> / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>> / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>> / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>> / - >>>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>> \ | >>>>>>> / - \ | / - \ | / - \ | >>>>>>> [?25h Stored in directory: >>>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1 >>>>>>> Successfully built future grpcio >>>>>>> Installing collected packages: six, grpcio, setuptools, protobuf, >>>>>>> grpcio-tools, future, mypy-protobuf >>>>>>> Successfully installed future-0.16.0 grpcio-1.28.1 >>>>>>> grpcio-tools-1.14.2 mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 >>>>>>> six-1.14.0 >>>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds. >>>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output >>>>>>> files). >>>>>>> Process Process-1: >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", >>>>>>> line >>>>>>> 292, in generate_proto_files >>>>>>> from grpc_tools import protoc >>>>>>> ModuleNotFoundError: No module named 'grpc_tools' >>>>>>> >>>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut. >>>>>>>> - If you find any important issues that you think should be >>>>>>>> addressed in the release, please tag the jira with fix version 2.21.0 >>>>>>>> and >>>>>>>> cc me (username `ibzib`). >>>>>>>> - Make sure to update the change log [2] with any significant >>>>>>>> changes if you haven't already. Send a PR with the change and tag me. >>>>>>>> (I >>>>>>>> imagine I'm not the only one who forgot to do this :).) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Kyle >>>>>>>> >>>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0 >>>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md >>>>>>>> >>>>>>>
