[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=380366&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-380366
]
ASF GitHub Bot logged work on BEAM-8979:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Feb/20 17:55
Start Date: 01/Feb/20 17:55
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10734: [BEAM-8979]
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581053581
> I think the phrases are limited to Apache committers sadly. Not sure if
this requirement will ever go away.
Understood. I'm going to try to apply the necessary pressure at work to get
CLA approved this week.
> Scan of subsequent run: https://gradle.com/s/nobazytmko36c
> It fails at the sdist task. Perhaps your shell already has this binary
installed, which is why it works for you.
I tested this locally. I removed `mypy_protobuf` from the venv that I'm
running gradle from, deleted the gradleenv directory, and ran `./gradlew
:sdks:python:sdist` and it worked fine for me locally. The gradleenv gets
recreated and `mypy_protobuf` gets pip installed into it.
Here's the output that I get (after fixing the logging in the latest commit):
```
> Task :sdks:python:setupVirtualenv
Already using interpreter
/Users/chad/dev/beam-tests/.venv-dev-typing/bin/python2.7
Using real prefix
'/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7'
New python executable in
/Users/chad/dev/beam-tests/beam-typing/build/gradleenv/1922375555/bin/python2.7
Also creating executable in
/Users/chad/dev/beam-tests/beam-typing/build/gradleenv/1922375555/bin/python
Installing setuptools, pip, wheel...
done.
Looking in links: file:///Users/chad/.cache/pip/wheelhouse
Collecting tox==3.11.1
Using cached tox-3.11.1-py2.py3-none-any.whl (76 kB)
Collecting grpcio-tools<=1.14.2,>=1.3.5
Using cached grpcio_tools-1.14.2-cp27-cp27m-macosx_10_12_x86_64.whl (1.8
MB)
Processing
/Users/chad/Library/Caches/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf15b1a/future-0.16.0-cp27-none-any.whl
Processing
/Users/chad/Library/Caches/pip/wheels/6e/95/9a/c57fed4950861038248dd0507909adc5661ffdd1887cb74ae2/mypy_protobuf-1.12-cp27-none-any.whl
Collecting six<2,>=1.0.0
Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Processing
/Users/chad/Library/Caches/pip/wheels/66/13/60/ef107438d90e4aad6320e3424e50cfce5e16d1e9aad6d38294/filelock-3.0.12-cp27-none-any.whl
Collecting virtualenv>=14.0.0
Using cached virtualenv-16.7.9-py2.py3-none-any.whl (3.4 MB)
Collecting pluggy<1,>=0.3.0
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting py<2,>=1.4.17
Using cached py-1.8.1-py2.py3-none-any.whl (83 kB)
Collecting toml>=0.9.4
Using cached toml-0.10.0-py2.py3-none-any.whl (25 kB)
Requirement already satisfied, skipping upgrade: setuptools>=30.0.0 in
/Users/chad/dev/beam-tests/beam-typing/build/gradleenv/1922375555/lib/python2.7/site-packages
(from tox==3.11.1) (44.0.0)
Collecting grpcio>=1.14.2
Using cached grpcio-1.26.0-cp27-cp27m-macosx_10_9_x86_64.whl (2.3 MB)
Collecting protobuf>=3.5.0.post1
Using cached protobuf-3.11.2-cp27-cp27m-macosx_10_9_x86_64.whl (1.3 MB)
Collecting importlib-metadata>=0.12; python_version < "3.8"
Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting enum34>=1.0.4; python_version < "3.4"
Using cached enum34-1.1.6-py2-none-any.whl (12 kB)
Collecting futures>=2.2.0; python_version < "3.2"
Using cached futures-3.3.0-py2-none-any.whl (16 kB)
Collecting contextlib2; python_version < "3"
Using cached contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)
Collecting pathlib2; python_version < "3"
Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Collecting zipp>=0.5
Using cached zipp-1.1.0-py2.py3-none-any.whl (4.6 kB)
Collecting configparser>=3.5; python_version < "3"
Using cached configparser-4.0.2-py2.py3-none-any.whl (22 kB)
Processing
/Users/chad/Library/Caches/pip/wheels/91/95/75/19c98a91239878abbc7c59970abd3b4e0438a7dd5b61778335/scandir-1.10.0-cp27-cp27m-macosx_10_13_x86_64.whl
Installing collected packages: six, filelock, virtualenv, contextlib2,
scandir, pathlib2, zipp, configparser, importlib-metadata, pluggy, py, toml,
tox, enum34, futures, grpcio, protobuf, grpcio-tools, future, mypy-protobuf
Successfully installed configparser-4.0.2 contextlib2-0.6.0.post1
enum34-1.1.6 filelock-3.0.12 future-0.16.0 futures-3.3.0 grpcio-1.26.0
grpcio-tools-1.14.2 importlib-metadata-1.5.0 mypy-protobuf-1.12 pathlib2-2.3.5
pluggy-0.13.1 protobuf-3.11.2 py-1.8.1 scandir-1.10.0 six-1.14.0 toml-0.10.0
tox-3.11.1 virtualenv-16.7.9 zipp-1.1.0
> Task :sdks:python:sdist
INFO:gen_protos:Found protoc_gen_mypy at
/Users/chad/dev/beam-tests/beam-typing/build/gradleenv/1922375555/bin/protoc-gen-mypy
...
```
What I noticed when looking at the [full log from the gradle
scan](https://scans.gradle.com/s/nobazytmko36c/console-log?anchor=73) is this:
```
:sdks:python:setupVirtualenv UP-TO-DATE
:sdks:python:sdist FAILED
```
That's `setupVirtualenv` task is supposed to install
`build-requirements.txt` which now contains the new `mypy_protobuf`
requirement. If that task is cached and doesn't run then that would explain
why it's not being installed. Do you think that could be the case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 380366)
Time Spent: 4h (was: 3h 50m)
> protoc-gen-mypy: program not found or is not executable
> -------------------------------------------------------
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
> Reporter: Kamil Wasilewski
> Assignee: Chad Dombrova
> Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in <module>
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
