The error you posted in your first log is very strange. pip is not able to
find one of its own vendored libraries, which is itself a very common dep
of our packages. I can't even think of what would cause that. You're
installing a bunch of unnecessary things before you run tox (tox will
handle installing those deps). Try it again without that. Here's what I
get:
chads-mbp-2:~/dev/beam-tests/beam-typing/sdks/python (typing-portability)
chad$ python3 -m venv .venv-tmp
chads-mbp-2:~/dev/beam-tests/beam-typing/sdks/python (typing-portability)
chad$ . .venv-tmp/bin/activate
(.venv-tmp) chads-mbp-2:~/dev/beam-tests/beam-typing/sdks/python
(typing-portability) chad$ pip install tox
Looking in links: file:///Users/chad/.cache/pip/wheelhouse
Collecting tox
Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB)
Collecting six>=1.14.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting
virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0
Downloading virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB)
|████████████████████████████████| 4.9 MB 2.2 MB/s
Collecting toml>=0.9.4
Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting py>=1.4.17
Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting filelock>=3.0.0
Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting packaging>=14
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting pluggy>=0.12.0
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting distlib<1,>=0.3.1
Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting appdirs<2,>=1.4.3
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting pyparsing>=2.0.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Installing collected packages: six, distlib, appdirs, filelock, virtualenv,
toml, py, pyparsing, packaging, pluggy, tox
Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12
packaging-20.4 pluggy-0.13.1 py-1.9.0 pyparsing-2.4.7 six-1.15.0
toml-0.10.2 tox-3.20.1 virtualenv-20.1.0
WARNING: You are using pip version 20.1.1; however, version 20.2.4 is
available.
You should consider upgrading via the
'/Users/chad/dev/beam-tests/beam-typing/sdks/python/.venv-tmp/bin/python3
-m pip install --upgrade pip' command.
(.venv-tmp) chads-mbp-2:~/dev/beam-tests/beam-typing/sdks/python
(typing-portability) chad$ tox -e py38
GLOB sdist-make: /Users/chad/dev/beam-tests/beam-typing/sdks/python/setup.py
py38 create:
/Users/chad/dev/beam-tests/beam-typing/sdks/python/target/.tox/py38
py38 installdeps: -rbuild-requirements.txt
py38 inst:
/Users/chad/dev/beam-tests/beam-typing/sdks/python/target/.tox/.tmp/package/1/apache-beam-2.26.0.dev0.zip
py38 installed: -f file:///Users/chad/.cache/pip/wheelhouse,apache-beam @
file:///Users/chad/dev/beam-tests/beam-typing/sdks/python/target/.tox/.tmp/package/1/apache-beam-2.26.0.dev0.zip,apipkg==1.5,atomicwrites==1.4.0,attrs==20.2.0,avro-python3==1.9.2.1,blindspin==2.0.1,certifi==2020.6.20,chardet==3.0.4,colorama==0.4.4,crayons==0.4.0,crcmod==1.7,deprecation==2.1.0,dill==0.3.1.1,docker==4.3.1,docopt==0.6.2,execnet==1.7.1,fastavro==1.1.0,freezegun==1.0.0,future==0.18.2,grpcio==1.33.2,grpcio-tools==1.30.0,hdfs==2.5.8,httplib2==0.17.4,idna==2.10,mock==2.0.0,more-itertools==8.6.0,mypy-protobuf==1.18,nose==1.3.7,nose-xunitmp==0.4.1,numpy==1.19.4,oauth2client==4.1.3,packaging==20.4,pandas==1.1.4,parameterized==0.7.4,pbr==5.5.1,pluggy==0.13.1,protobuf==3.13.0,psycopg2-binary==2.8.6,py==1.9.0,pyarrow==0.17.1,pyasn1==0.4.8,pyasn1-modules==0.2.8,pydot==1.4.1,PyHamcrest==1.10.1,pymongo==3.11.0,pyparsing==2.4.7,pytest==4.6.11,pytest-forked==1.3.0,pytest-timeout==1.4.2,pytest-xdist==1.34.0,python-dateutil==2.8.1,pytz==2020.4,PyYAML==5.3.1,requests==2.24.0,requests-mock==1.8.0,rsa==4.6,six==1.15.0,SQLAlchemy==1.3.20,tenacity==5.1.5,testcontainers==3.1.0,typing-extensions==3.7.4.3,urllib3==1.25.11,wcwidth==0.2.5,websocket-client==0.57.0,wrapt==1.12.1
py38 run-test-pre: PYTHONHASHSEED='246162265'
py38 run-test-pre: commands[0] | python --version
Python 3.8.5
py38 run-test-pre: commands[1] | pip --version
pip 20.2.3 from
/Users/chad/dev/beam-tests/beam-typing/sdks/python/target/.tox/py38/lib/python3.8/site-packages/pip
(python 3.8)
py38 run-test-pre: commands[2] | pip check
No broken requirements found.
py38 run-test-pre: commands[3] | bash
/Users/chad/dev/beam-tests/beam-typing/sdks/python/scripts/run_tox_cleanup.sh
py38 run-test: commands[0] | python
apache_beam/examples/complete/autocomplete_test.py
s.
----------------------------------------------------------------------
Ran 2 tests in 0.627s
OK (skipped=1)
py38 run-test: commands[1] |
/Users/chad/dev/beam-tests/beam-typing/sdks/python/scripts/run_pytest.sh
py38 ''
=================================================================================
test session starts
==================================================================================
platform darwin -- Python 3.8.5, pytest-4.6.11, py-1.9.0, pluggy-0.13.1
cachedir: target/.tox/py38/.pytest_cache
rootdir: /Users/chad/dev/beam-tests/beam-typing/sdks/python, inifile:
pytest.ini
plugins: requests-mock-1.8.0, xdist-1.34.0, timeout-1.4.2, forked-1.3.0
timeout: 600.0s
timeout method: signal
timeout func_only: False
gw0 [3877] / gw1 [3877] / gw2 [3877] / gw3 [3877] / gw4 [3877] / gw5 [3877]
On Wed, Nov 4, 2020 at 10:17 PM Alex Amato <ajam...@google.com> wrote:
> Well, running as sudo just hits a different error
>
> (my-virtual-env-3.6.10) ajamato@ajamato-linux0:~/beam/sdks/python$ sudo
> tox -e py36-lint
> [sudo] password for ajamato:
> GLOB sdist-make: /usr/local/google/home/ajamato/beam/sdks/python/setup.py
> py36-lint create:
> /usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint
> ERROR: invocation failed (exit code 1), logfile:
> /usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/log/py36-lint-0.log
> =======================================================================================================================================
> log start
> ========================================================================================================================================
> RuntimeError: failed to query /usr/bin/python3.6 with code 1 err:
> 'Traceback (most recent call last):\n File
> "/usr/lib/python3/dist-packages/virtualenv/discovery/py_info.py", line 16,
> in <module>\n from distutils import dist\nImportError: cannot import
> name \'dist\'\n'
>
> ========================================================================================================================================
> log end
> =========================================================================================================================================
> ERROR: InvocationError for command /usr/bin/python3 -m virtualenv
> --no-download --python /usr/bin/python3.6 py36-lint (exited with code 1)
> ________________________________________________________________________________________________________________________________________
> summary
> _________________________________________________________________________________________________________________________________________
> ERROR: py36-lint: InvocationError for command /usr/bin/python3 -m
> virtualenv --no-download --python /usr/bin/python3.6 py36-lint (exited with
> code 1)
>
> On Wed, Nov 4, 2020 at 10:16 PM Alex Amato <ajam...@google.com> wrote:
>
>> I see, well. I have setup a new virutalenv using pyenv for python 3.6.10
>>
>> Then ran these steps from the python tips guide+tox+pytest install
>>
>> # Install setup.py requirements.
>> (env) $ pip install -r build-requirements.txt
>>
>> # Install packages.
>> (env) $ pip install -e .[gcp,test]
>>
>> (env) $ pip install pytest
>>
>> (env) $ pip install tox
>> Tried running tox -e py36-lint and tox -e py37-lint
>>
>>
>> (Realized that I might only be able to run py36-lint from my 3.6.10
>> environment? but not sure)
>> Got this for py36-lint
>>
>> =======================================================================================================================================
>> log start
>> ========================================================================================================================================
>> ERROR: invocation failed (exit code 1), logfile:
>> /usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/log/py36-lint-1.log
>> =======================================================================================================================================
>> log start
>> ========================================================================================================================================
>> Traceback (most recent call last):
>> File "target/.tox/py36-lint/bin/pip", line 5, in <module>
>> from pip._internal.cli.main import main
>> File
>> "/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/lib/python3.6/site-packages/pip/_internal/cli/main.py",
>> line 10, in <module>
>> from pip._internal.cli.autocompletion import autocomplete
>> File
>> "/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py",
>> line 9, in <module>
>> from pip._internal.cli.main_parser import create_main_parser
>> File
>> "/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py",
>> line 7, in <module>
>> from pip._internal.cli import cmdoptions
>> File
>> "/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/lib/python3.6/site-packages/pip/_internal/cli/cmdoptions.py",
>> line 24, in <module>
>> from pip._internal.cli.progress_bars import BAR_TYPES
>> File
>> "/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/lib/python3.6/site-packages/pip/_internal/cli/progress_bars.py",
>> line 7, in <module>
>> from pip._vendor import six
>> ImportError: cannot import name 'six'
>>
>> ========================================================================================================================================
>> log end
>> =========================================================================================================================================
>> ERROR: could not install deps [-rbuild-requirements.txt]; v =
>> InvocationError('/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/bin/python
>> target/.tox/py36-lint/bin/pip install --retries 10
>> -rbuild-requirements.txt', 1)
>> ________________________________________________________________________________________________________________________________________
>> summary
>> _________________________________________________________________________________________________________________________________________
>> ERROR: py36-lint: could not install deps [-rbuild-requirements.txt]; v
>> =
>> InvocationError('/usr/local/google/home/ajamato/beam/sdks/python/target/.tox/py36-lint/bin/python
>> target/.tox/py36-lint/bin/pip install --retries 10
>> -rbuild-requirements.txt', 1)
>> ======
>>
>> tried pip install six,as well, but I am met with
>> Requirement already satisfied: six in
>> /usr/local/google/home/ajamato/.pyenv/versions/3.6.10/envs/my-virtual-env-3.6.10/lib/python3.6/site-packages
>> (1.15.0)
>>
>>
>>
>> I am guessing something is preventing tox from doing some steps? Does one
>> normally run tox under sudo?
>>
>>
>> On Wed, Nov 4, 2020 at 10:05 PM Chad Dombrova <chad...@gmail.com> wrote:
>>
>>>
>>> All of these are great suggestions. I think what I really need though is
>>>> some way to figure out how to cleanly install (perhaps reinstalling)
>>>> everything I need to run all these commands. tox, yapf,
>>>>
>>>
>>> tox should be the only thing you need to install. After that, tox will
>>> install whatever you need to run the tests. pre-commit accomplishes
>>> something similar, but just for the pre-commit git hooks.
>>>
>>> -chad
>>>
>>>
