Thanks for the fix Max. On Mon, May 11, 2020 at 5:46 AM Maximilian Michels <[email protected]> wrote:
> FYI I've created this issue and marked it as a blocker: > https://jira.apache.org/jira/browse/BEAM-9947 > > Essentially, the timer encoding is broken for all non-standard key > coders. The fix can be found here: > https://github.com/apache/beam/pull/11658 > > -Max > > On 08.05.20 18:53, Udi Meiri wrote: > > +Chad Dombrova <mailto:[email protected]> , who added > _find_protoc_gen_mypy. > > > > I'm guessing that the code > > in _install_grpcio_tools_and_generate_proto_files creates a kind of > > virtualenv, but it only works well for staging Python modules and not > > binaries like protoc-gen-mypy. > > (I assume there's a reason why it doesn't invoke virtualenv, probably > > since the list of things setup.py can expect to be installed is very > > minimal (setuptools).) > > > > One solution would be to make these setup.py dependencies explicit in > > pyproject.toml, such that pip installs them before running > > setup.py: > https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support > > It would help when using tools like pip ("pip wheel"), but I'm not sure > > what the alternative for "python setup.py sdist" is. > > > > > > On Thu, May 7, 2020 at 10:40 PM Thomas Weise <[email protected] > > <mailto:[email protected]>> wrote: > > > > No additional stacktraces. Full error output below. > > > > It's not clear what is going wrong. > > > > There isn't any exception from the subprocess execution since the > > "WARNING:root:Installing grpcio-tools took 305.39 seconds." is > printed. > > > > Also, the time it takes to perform the install is equivalent to > > successfully running the pip command. > > > > I will report back if I find anything else. Currently doing the > > explicit install via pip install -r > sdks/python/build-requirements.txt > > > > Thanks, > > Thomas > > > > WARNING:root:Installing grpcio-tools took 269.27 seconds. > > INFO:gen_protos:Regenerating Python proto definitions (no output > files). > > Process Process-1: > > Traceback (most recent call last): > > File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in > > _bootstrap > > self.run() > > File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in > run > > self._target(*self._args, **self._kwargs) > > File > > > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > > 378, in _install_grpcio_tools_and_generate_proto_files > > generate_proto_files(force=force) > > File > > > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > > 315, in generate_proto_files > > protoc_gen_mypy = _find_protoc_gen_mypy() > > File > > > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > > 233, in _find_protoc_gen_mypy > > (fname, ', '.join(search_paths))) > > RuntimeError: Could not find protoc-gen-mypy in > > /code/venvs/venv2/bin, /code/venvs/venv2/bin, /code/venvs/venv3/bin, > > /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin > > Traceback (most recent call last): > > File "setup.py", line 311, in <module> > > 'mypy': generate_protos_first(mypy), > > File > > > "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py", > > line 129, in setup > > return distutils.core.setup(**attrs) > > File "/usr/lib/python2.7/distutils/core.py", line 151, in setup > > dist.run_commands() > > File "/usr/lib/python2.7/distutils/dist.py", line 953, in > run_commands > > self.run_command(cmd) > > File "/usr/lib/python2.7/distutils/dist.py", line 972, in > run_command > > cmd_obj.run() > > File > > > "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", > > line 204, in run > > self.run_command('build') > > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in > run_command > > self.distribution.run_command(command) > > File "/usr/lib/python2.7/distutils/dist.py", line 972, in > run_command > > cmd_obj.run() > > File "/usr/lib/python2.7/distutils/command/build.py", line 128, in > run > > self.run_command(cmd_name) > > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in > run_command > > self.distribution.run_command(command) > > File "/usr/lib/python2.7/distutils/dist.py", line 972, in > run_command > > cmd_obj.run() > > File "setup.py", line 235, in run > > gen_protos.generate_proto_files() > > File > > > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > > 310, in generate_proto_files > > raise ValueError("Proto generation failed (see log for > details).") > > ValueError: Proto generation failed (see log for details). > > > > > > On Thu, May 7, 2020 at 2:25 PM Udi Meiri <[email protected] > > <mailto:[email protected]>> wrote: > > > > It's hard to say without more details what's going on. Ahmet > > you're right that it installs build-requirements.txt and retries > > calling generate_proto_files(). > > > > Thomas, were there additional stacktraces? (after a "During > > handling of the above exception, another exception occurred:" > > message?) > > > > > > On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <[email protected] > > <mailto:[email protected]>> wrote: > > > > > > > > On Thu, May 7, 2020 at 11:56 AM Thomas Weise <[email protected] > > <mailto:[email protected]>> wrote: > > > > Thanks Udi! This is the issue. I'm trying to upgrade > > from 2.18 where build-requirements.txt didn't exist. > > > > Is there a reason why this cannot happen automatically > > when running python3.6 setup.py sdist bdist_wheel ? > > > > > > I _believe_ this should happen automatically > > here: > https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365. > > Maybe there is a problem there? > > > > > > > > Thomas > > > > > > On Thu, May 7, 2020 at 11:07 AM Udi Meiri > > <[email protected] <mailto:[email protected]>> wrote: > > > > Probably not the issue, but double checking: are you > > running "pip install -r > > sdks/python/build-requirements.txt" first? > > > > On Wed, May 6, 2020 at 7:22 PM Thomas Weise > > <[email protected] <mailto:[email protected]>> wrote: > > > > I'm working on rebasing our fork to 2.21.0 and > > run into a problem installing grpcio-tools that > > leads to *ModuleNotFoundError: No module named > > 'grpc_tools' *(see details below) > > > > I cannot reproduce this locally. > > > > Any =suggestions on what to look for? > > > > Thanks, > > Thomas > > > > [?25hBuilding wheels for collected packages: > > future, gr > > Running setup.py bdist_wheel for future ... > > [?25l- \ > > [?25h Stored in directory: > > > > /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1 > > Running setup.py bdist_wheel for grpcio ... > > [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > > | / - \ | / - \ | / - \ | > > [?25h Stored in directory: > > > > /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1 > > Successfully built future grpcio > > Installing collected packages: six, grpcio, > > setuptools, protobuf, grpcio-tools, future, > > mypy-protobuf > > Successfully installed future-0.16.0 > > grpcio-1.28.1 grpcio-tools-1.14.2 > > mypy-protobuf-1.18 protobuf-3.11.3 > > setuptools-46.1.3 six-1.14.0 > > WARNING:root:Installing grpcio-tools took 305.39 > > seconds. > > INFO:gen_protos:Regenerating Python proto > > definitions (no output files). > > Process Process-1: > > Traceback (most recent call last): > > File > > > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", > > line 292, in generate_proto_files > > from grpc_tools import protoc > > ModuleNotFoundError: No module named 'grpc_tools' > > > > On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi everyone, > > > > Just a heads up that the Beam 2.21 release > > branch [1] is cut. > > - If you find any important issues that you > > think should be addressed in the release, > > please tag the jira with fix version 2.21.0 > > and cc me (username `ibzib`). > > - Make sure to update the change log [2] > > with any significant changes if you haven't > > already. Send a PR with the change and tag > > me. (I imagine I'm not the only one who > > forgot to do this :).) > > > > Thanks, > > Kyle > > > > [1] > https://github.com/apache/beam/blob/release-2.21.0 > > [2] > https://github.com/apache/beam/blob/master/CHANGES.md > > >
