Thanks for the heads up Thomas. Please let us know as soon as possible what you find.
On Fri, May 8, 2020 at 2:43 PM Thomas Weise <[email protected]> wrote: > I could not find a way for the hidden pip install to succeed within the > virtual environment. > > Also, heads up that we found another issue related to timer encoding that > looks like a release blocker. Details to follow. > > > On Fri, May 8, 2020 at 9:53 AM Udi Meiri <[email protected]> wrote: > >> +Chad Dombrova <[email protected]> , who added _find_protoc_gen_mypy. >> >> I'm guessing that the code >> in _install_grpcio_tools_and_generate_proto_files creates a kind of >> virtualenv, but it only works well for staging Python modules and not >> binaries like protoc-gen-mypy. >> (I assume there's a reason why it doesn't invoke virtualenv, probably >> since the list of things setup.py can expect to be installed is very >> minimal (setuptools).) >> >> One solution would be to make these setup.py dependencies explicit in >> pyproject.toml, such that pip installs them before running setup.py: >> https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support >> It would help when using tools like pip ("pip wheel"), but I'm not sure >> what the alternative for "python setup.py sdist" is. >> >> >> On Thu, May 7, 2020 at 10:40 PM Thomas Weise <[email protected]> wrote: >> >>> No additional stacktraces. Full error output below. >>> >>> It's not clear what is going wrong. >>> >>> There isn't any exception from the subprocess execution since the >>> "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed. >>> >>> Also, the time it takes to perform the install is equivalent to >>> successfully running the pip command. >>> >>> I will report back if I find anything else. Currently doing the >>> explicit install via pip install -r sdks/python/build-requirements.txt >>> >>> Thanks, >>> Thomas >>> >>> WARNING:root:Installing grpcio-tools took 269.27 seconds. >>> INFO:gen_protos:Regenerating Python proto definitions (no output files). >>> Process Process-1: >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in >>> _bootstrap >>> self.run() >>> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run >>> self._target(*self._args, **self._kwargs) >>> File >>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >>> 378, in _install_grpcio_tools_and_generate_proto_files >>> generate_proto_files(force=force) >>> File >>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >>> 315, in generate_proto_files >>> protoc_gen_mypy = _find_protoc_gen_mypy() >>> File >>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >>> 233, in _find_protoc_gen_mypy >>> (fname, ', '.join(search_paths))) >>> RuntimeError: Could not find protoc-gen-mypy in /code/venvs/venv2/bin, >>> /code/venvs/venv2/bin, /code/venvs/venv3/bin, /usr/local/sbin, >>> /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin >>> Traceback (most recent call last): >>> File "setup.py", line 311, in <module> >>> 'mypy': generate_protos_first(mypy), >>> File >>> "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py", >>> line 129, in setup >>> return distutils.core.setup(**attrs) >>> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup >>> dist.run_commands() >>> File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands >>> self.run_command(cmd) >>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command >>> cmd_obj.run() >>> File >>> "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", >>> line 204, in run >>> self.run_command('build') >>> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command >>> self.distribution.run_command(command) >>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command >>> cmd_obj.run() >>> File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run >>> self.run_command(cmd_name) >>> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command >>> self.distribution.run_command(command) >>> File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command >>> cmd_obj.run() >>> File "setup.py", line 235, in run >>> gen_protos.generate_proto_files() >>> File >>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line >>> 310, in generate_proto_files >>> raise ValueError("Proto generation failed (see log for details).") >>> ValueError: Proto generation failed (see log for details). >>> >>> >>> On Thu, May 7, 2020 at 2:25 PM Udi Meiri <[email protected]> wrote: >>> >>>> It's hard to say without more details what's going on. Ahmet you're >>>> right that it installs build-requirements.txt and retries calling >>>> generate_proto_files(). >>>> >>>> Thomas, were there additional stacktraces? (after a "During handling of >>>> the above exception, another exception occurred:" message?) >>>> >>>> >>>> On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Thu, May 7, 2020 at 11:56 AM Thomas Weise <[email protected]> wrote: >>>>> >>>>>> Thanks Udi! This is the issue. I'm trying to upgrade from 2.18 where >>>>>> build-requirements.txt didn't exist. >>>>>> >>>>>> Is there a reason why this cannot happen automatically when >>>>>> running python3.6 setup.py sdist bdist_wheel ? >>>>>> >>>>> >>>>> I _believe_ this should happen automatically here: >>>>> https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365. >>>>> Maybe there is a problem there? >>>>> >>>>> >>>>>> >>>>>> Thomas >>>>>> >>>>>> >>>>>> On Thu, May 7, 2020 at 11:07 AM Udi Meiri <[email protected]> wrote: >>>>>> >>>>>>> Probably not the issue, but double checking: are you running "pip >>>>>>> install -r sdks/python/build-requirements.txt" first? >>>>>>> >>>>>>> On Wed, May 6, 2020 at 7:22 PM Thomas Weise <[email protected]> wrote: >>>>>>> >>>>>>>> I'm working on rebasing our fork to 2.21.0 and run into a problem >>>>>>>> installing grpcio-tools that leads to *ModuleNotFoundError: No >>>>>>>> module named 'grpc_tools' *(see details below) >>>>>>>> >>>>>>>> I cannot reproduce this locally. >>>>>>>> >>>>>>>> Any =suggestions on what to look for? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Thomas >>>>>>>> >>>>>>>> [?25hBuilding wheels for collected packages: future, gr >>>>>>>> Running setup.py bdist_wheel for future ... [?25l- \ >>>>>>>> [?25h Stored in directory: >>>>>>>> /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1 >>>>>>>> Running setup.py bdist_wheel for grpcio ... [?25l- \ | / - \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>>> \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>>> \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>>> \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>>> \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>>> \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - >>>>>>>> \ | / >>>>>>>> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | >>>>>>>> / - \ >>>>>>>> | / - \ | / - \ | / - \ | >>>>>>>> [?25h Stored in directory: >>>>>>>> /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1 >>>>>>>> Successfully built future grpcio >>>>>>>> Installing collected packages: six, grpcio, setuptools, protobuf, >>>>>>>> grpcio-tools, future, mypy-protobuf >>>>>>>> Successfully installed future-0.16.0 grpcio-1.28.1 >>>>>>>> grpcio-tools-1.14.2 mypy-protobuf-1.18 protobuf-3.11.3 >>>>>>>> setuptools-46.1.3 >>>>>>>> six-1.14.0 >>>>>>>> WARNING:root:Installing grpcio-tools took 305.39 seconds. >>>>>>>> INFO:gen_protos:Regenerating Python proto definitions (no output >>>>>>>> files). >>>>>>>> Process Process-1: >>>>>>>> Traceback (most recent call last): >>>>>>>> File >>>>>>>> "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", >>>>>>>> line >>>>>>>> 292, in generate_proto_files >>>>>>>> from grpc_tools import protoc >>>>>>>> ModuleNotFoundError: No module named 'grpc_tools' >>>>>>>> >>>>>>>> On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi everyone, >>>>>>>>> >>>>>>>>> Just a heads up that the Beam 2.21 release branch [1] is cut. >>>>>>>>> - If you find any important issues that you think should be >>>>>>>>> addressed in the release, please tag the jira with fix version 2.21.0 >>>>>>>>> and >>>>>>>>> cc me (username `ibzib`). >>>>>>>>> - Make sure to update the change log [2] with any significant >>>>>>>>> changes if you haven't already. Send a PR with the change and tag me. >>>>>>>>> (I >>>>>>>>> imagine I'm not the only one who forgot to do this :).) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Kyle >>>>>>>>> >>>>>>>>> [1] https://github.com/apache/beam/blob/release-2.21.0 >>>>>>>>> [2] https://github.com/apache/beam/blob/master/CHANGES.md >>>>>>>>> >>>>>>>>
