+Chad Dombrova <chad...@gmail.com> , who added _find_protoc_gen_mypy.
I'm guessing that the code in _install_grpcio_tools_and_generate_proto_files creates a kind of virtualenv, but it only works well for staging Python modules and not binaries like protoc-gen-mypy. (I assume there's a reason why it doesn't invoke virtualenv, probably since the list of things setup.py can expect to be installed is very minimal (setuptools).) One solution would be to make these setup.py dependencies explicit in pyproject.toml, such that pip installs them before running setup.py: https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support It would help when using tools like pip ("pip wheel"), but I'm not sure what the alternative for "python setup.py sdist" is. On Thu, May 7, 2020 at 10:40 PM Thomas Weise <t...@apache.org> wrote: > No additional stacktraces. Full error output below. > > It's not clear what is going wrong. > > There isn't any exception from the subprocess execution since the > "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed. > > Also, the time it takes to perform the install is equivalent to > successfully running the pip command. > > I will report back if I find anything else. Currently doing the > explicit install via pip install -r sdks/python/build-requirements.txt > > Thanks, > Thomas > > WARNING:root:Installing grpcio-tools took 269.27 seconds. > INFO:gen_protos:Regenerating Python proto definitions (no output files). > Process Process-1: > Traceback (most recent call last): > File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in > _bootstrap > self.run() > File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 378, in _install_grpcio_tools_and_generate_proto_files > generate_proto_files(force=force) > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 315, in generate_proto_files > protoc_gen_mypy = _find_protoc_gen_mypy() > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 233, in _find_protoc_gen_mypy > (fname, ', '.join(search_paths))) > RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin, > /code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin, > /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin > Traceback (most recent call last): > File "setup.py", line 311, in <module> > 'mypy': generate_protos_first(mypy), > File > "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py", > line 129, in setup > return distutils.core.setup(**attrs) > File "/usr/lib/python2.7/distutils/core.py", line 151, in setup > dist.run_commands() > File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands > self.run_command(cmd) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File > "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", > line 204, in run > self.run_command('build') > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command > self.distribution.run_command(command) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run > self.run_command(cmd_name) > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command > self.distribution.run_command(command) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "setup.py", line 235, in run > gen_protos.generate_proto_files() > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 310, in generate_proto_files > raise ValueError("Proto generation failed (see log for details).") > ValueError: Proto generation failed (see log for details). > > > On Thu, May 7, 2020 at 2:25 PM Udi Meiri <eh...@google.com> wrote: > >> It's hard to say without more details what's going on. Ahmet you're right >> that it installs build-requirements.txt and retries calling >> generate_proto_files(). >> >> Thomas, were there additional stacktraces? (after a "During handling of >> the above exception, another exception occurred:" message?) >> >> >> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com> wrote: >> >>> >>> >>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <t...@apache.org> wrote: >>> >>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where >>>> build-requirements.txt didn't exist. >>>> >>>> Is there a reason why this cannot happen automatically when >>>> running python3.6 setup.py sdist bdist_wheel ? >>>> >>> >>> I _believe_ this should happen automatically here: >>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365. >>> Maybe there is a problem there? >>> >>> >>>> >>>> Thomas >>>> >>>> >>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <eh...@google.com> wrote: >>>> >>>>> Probably not the issue, but double checking: are you running "pip >>>>> install -r sdks/python/build-requirements.txt" first? >>>>> >>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <t...@apache.org> wrote: >>>>> >>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem >>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No >>>>>> module named 'grpc_tools' *(see details below) >>>>>> >>>>>> I cannot reproduce this locally. >>>>>> >>>>>> Any =suggestions on what to look for? >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>> [?25hBuilding wheels for collected packages: future, gr >>>>>> Running setup.py bdist_wheel for future ... [?25l- \ >>>>>> [?25h Stored in directory: >>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1 >>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>> / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>> / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>> / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>> / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>> / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>> / - >>>>>> \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>> \ | >>>>>> / - \ | / - \ | / - \ | >>>>>> [?25h Stored in directory: >>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1 >>>>>> Successfully built future grpcio >>>>>> Installing collected packages: six, grpcio, setuptools, protobuf, >>>>>> grpcio-tools, future, mypy-protobuf >>>>>> Successfully installed future-0.16.0 grpcio-1.28.1 >>>>>> grpcio-tools-1.14.2 mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 >>>>>> six-1.14.0 >>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds. >>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output >>>>>> files). >>>>>> Process Process-1: >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", >>>>>> line >>>>>> 292, in generate_proto_files >>>>>> from grpc_tools import protoc >>>>>> ModuleNotFoundError: No module named 'grpc_tools' >>>>>> >>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <kcwea...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut. >>>>>>> - If you find any important issues that you think should be >>>>>>> addressed in the release, please tag the jira with fix version 2.21.0 >>>>>>> and >>>>>>> cc me (username `ibzib`). >>>>>>> - Make sure to update the change log [2] with any significant >>>>>>> changes if you haven't already. Send a PR with the change and tag me. (I >>>>>>> imagine I'm not the only one who forgot to do this :).) >>>>>>> >>>>>>> Thanks, >>>>>>> Kyle >>>>>>> >>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0 >>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md >>>>>>> >>>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
