Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Alex Amato]

Thanks, updated the JIRA with a link to this thread and a note of what
could be done.

On Mon, Apr 29, 2019 at 10:29 AM Udi Meiri  wrote:

> Pip has a --cache-dir which should be safe with concurrent writes.
>
> On Fri, Apr 26, 2019 at 3:59 PM Ahmet Altay  wrote:
>
>> It is possible to download dependencies with pip to a local directory and
>> install from there [1]. As a side benefit this is supposed to speed up the
>> installation process. Since we setup virtualenv multiple times, this could
>> actually help us in a single run. And if we can keep this cache across test
>> runs we can reduce flakiness.
>>
>> [1]
>> https://pip.pypa.io/en/latest/user_guide/#installing-from-local-packages
>>
>> On Fri, Apr 26, 2019 at 3:42 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> We do retry certain inherently flaky tests, for example, see[1]. This
>>> practice should be used with caution, see discussion [2].
>>>
>>> However retrying an individual test would not avoid the flake that Alex
>>> brought up in this thread, we'd have to retry setupVirtualEnv task that is
>>> executed once per suite of tests. Retrying just that task is different from
>>> retrying the whole suite.
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41,
>>> this was discussed
>>> [2]
>>> https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E
>>>  discussed
>>>
>>> On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri  wrote:
>>>
 Alex, I changed my mind: I'm okay retrying single tests, just not
 entire suites of tests (e.g. if precommits take an hour, retrying the run
 takes up an additional hour on the Jenkins machine).
 This is more of an issue in Python, where gradle does not (currently)
 have insight into which tests failed and how to retry just them.



 On Fri, Apr 26, 2019 at 2:17 PM Alex Amato  wrote:

> @Udi Meiri , Is this true if the specific tests are
> rerun? I don't think we should rerun all tests.
>
> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Preinstalling dependencies may affect the dependency resolution, and
>> we may end up testing a different configuration than a user would have
>> after installing beam into a clean environment.
>>
>> I do think pip uses cache, unless one specifies "--no-cache-dir". By
>> default the cache is ~/.cache/pip. Looking up the log message in OP, we 
>> can
>> see several "Using cached..." log entries. Not sure why futures was not
>> fetched from cache or PyPi. Perhaps it is also a pip flake.
>>
>> I would be against wiping flakes under the rug by rerunning the whole
>> suite after an error, but re-rerunning parts of the test environment set
>> up, that are prone to environmental flakes, such as setupVirtualEnv seems
>> reasonable. I agree with Udi that care should be taken to not overload
>> Jenkins (e.g. retries should be limited)
>>
>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Udi Meiri]

Pip has a --cache-dir which should be safe with concurrent writes.

On Fri, Apr 26, 2019 at 3:59 PM Ahmet Altay  wrote:

> It is possible to download dependencies with pip to a local directory and
> install from there [1]. As a side benefit this is supposed to speed up the
> installation process. Since we setup virtualenv multiple times, this could
> actually help us in a single run. And if we can keep this cache across test
> runs we can reduce flakiness.
>
> [1]
> https://pip.pypa.io/en/latest/user_guide/#installing-from-local-packages
>
> On Fri, Apr 26, 2019 at 3:42 PM Valentyn Tymofieiev 
> wrote:
>
>> We do retry certain inherently flaky tests, for example, see[1]. This
>> practice should be used with caution, see discussion [2].
>>
>> However retrying an individual test would not avoid the flake that Alex
>> brought up in this thread, we'd have to retry setupVirtualEnv task that is
>> executed once per suite of tests. Retrying just that task is different from
>> retrying the whole suite.
>>
>> [1]
>> https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41,
>> this was discussed
>> [2]
>> https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E
>>  discussed
>>
>> On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri  wrote:
>>
>>> Alex, I changed my mind: I'm okay retrying single tests, just not entire
>>> suites of tests (e.g. if precommits take an hour, retrying the run takes up
>>> an additional hour on the Jenkins machine).
>>> This is more of an issue in Python, where gradle does not (currently)
>>> have insight into which tests failed and how to retry just them.
>>>
>>>
>>>
>>> On Fri, Apr 26, 2019 at 2:17 PM Alex Amato  wrote:
>>>
 @Udi Meiri , Is this true if the specific tests are
 rerun? I don't think we should rerun all tests.

 On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Preinstalling dependencies may affect the dependency resolution, and
> we may end up testing a different configuration than a user would have
> after installing beam into a clean environment.
>
> I do think pip uses cache, unless one specifies "--no-cache-dir". By
> default the cache is ~/.cache/pip. Looking up the log message in OP, we 
> can
> see several "Using cached..." log entries. Not sure why futures was not
> fetched from cache or PyPi. Perhaps it is also a pip flake.
>
> I would be against wiping flakes under the rug by rerunning the whole
> suite after an error, but re-rerunning parts of the test environment set
> up, that are prone to environmental flakes, such as setupVirtualEnv seems
> reasonable. I agree with Udi that care should be taken to not overload
> Jenkins (e.g. retries should be limited)
>



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Ahmet Altay]

It is possible to download dependencies with pip to a local directory and
install from there [1]. As a side benefit this is supposed to speed up the
installation process. Since we setup virtualenv multiple times, this could
actually help us in a single run. And if we can keep this cache across test
runs we can reduce flakiness.

[1] https://pip.pypa.io/en/latest/user_guide/#installing-from-local-packages

On Fri, Apr 26, 2019 at 3:42 PM Valentyn Tymofieiev 
wrote:

> We do retry certain inherently flaky tests, for example, see[1]. This
> practice should be used with caution, see discussion [2].
>
> However retrying an individual test would not avoid the flake that Alex
> brought up in this thread, we'd have to retry setupVirtualEnv task that is
> executed once per suite of tests. Retrying just that task is different from
> retrying the whole suite.
>
> [1]
> https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41,
> this was discussed
> [2]
> https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E
>  discussed
>
> On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri  wrote:
>
>> Alex, I changed my mind: I'm okay retrying single tests, just not entire
>> suites of tests (e.g. if precommits take an hour, retrying the run takes up
>> an additional hour on the Jenkins machine).
>> This is more of an issue in Python, where gradle does not (currently)
>> have insight into which tests failed and how to retry just them.
>>
>>
>>
>> On Fri, Apr 26, 2019 at 2:17 PM Alex Amato  wrote:
>>
>>> @Udi Meiri , Is this true if the specific tests are
>>> rerun? I don't think we should rerun all tests.
>>>
>>> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Preinstalling dependencies may affect the dependency resolution, and we
 may end up testing a different configuration than a user would have after
 installing beam into a clean environment.

 I do think pip uses cache, unless one specifies "--no-cache-dir". By
 default the cache is ~/.cache/pip. Looking up the log message in OP, we can
 see several "Using cached..." log entries. Not sure why futures was not
 fetched from cache or PyPi. Perhaps it is also a pip flake.

 I would be against wiping flakes under the rug by rerunning the whole
 suite after an error, but re-rerunning parts of the test environment set
 up, that are prone to environmental flakes, such as setupVirtualEnv seems
 reasonable. I agree with Udi that care should be taken to not overload
 Jenkins (e.g. retries should be limited)

>>>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Valentyn Tymofieiev]

We do retry certain inherently flaky tests, for example, see[1]. This
practice should be used with caution, see discussion [2].

However retrying an individual test would not avoid the flake that Alex
brought up in this thread, we'd have to retry setupVirtualEnv task that is
executed once per suite of tests. Retrying just that task is different from
retrying the whole suite.

[1]
https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41,
this was discussed
[2]
https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E
 discussed

On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri  wrote:

> Alex, I changed my mind: I'm okay retrying single tests, just not entire
> suites of tests (e.g. if precommits take an hour, retrying the run takes up
> an additional hour on the Jenkins machine).
> This is more of an issue in Python, where gradle does not (currently) have
> insight into which tests failed and how to retry just them.
>
>
>
> On Fri, Apr 26, 2019 at 2:17 PM Alex Amato  wrote:
>
>> @Udi Meiri , Is this true if the specific tests are
>> rerun? I don't think we should rerun all tests.
>>
>> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Preinstalling dependencies may affect the dependency resolution, and we
>>> may end up testing a different configuration than a user would have after
>>> installing beam into a clean environment.
>>>
>>> I do think pip uses cache, unless one specifies "--no-cache-dir". By
>>> default the cache is ~/.cache/pip. Looking up the log message in OP, we can
>>> see several "Using cached..." log entries. Not sure why futures was not
>>> fetched from cache or PyPi. Perhaps it is also a pip flake.
>>>
>>> I would be against wiping flakes under the rug by rerunning the whole
>>> suite after an error, but re-rerunning parts of the test environment set
>>> up, that are prone to environmental flakes, such as setupVirtualEnv seems
>>> reasonable. I agree with Udi that care should be taken to not overload
>>> Jenkins (e.g. retries should be limited)
>>>
>>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Udi Meiri]

Alex, I changed my mind: I'm okay retrying single tests, just not entire
suites of tests (e.g. if precommits take an hour, retrying the run takes up
an additional hour on the Jenkins machine).
This is more of an issue in Python, where gradle does not (currently) have
insight into which tests failed and how to retry just them.



On Fri, Apr 26, 2019 at 2:17 PM Alex Amato  wrote:

> @Udi Meiri , Is this true if the specific tests are
> rerun? I don't think we should rerun all tests.
>
> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev 
> wrote:
>
>> Preinstalling dependencies may affect the dependency resolution, and we
>> may end up testing a different configuration than a user would have after
>> installing beam into a clean environment.
>>
>> I do think pip uses cache, unless one specifies "--no-cache-dir". By
>> default the cache is ~/.cache/pip. Looking up the log message in OP, we can
>> see several "Using cached..." log entries. Not sure why futures was not
>> fetched from cache or PyPi. Perhaps it is also a pip flake.
>>
>> I would be against wiping flakes under the rug by rerunning the whole
>> suite after an error, but re-rerunning parts of the test environment set
>> up, that are prone to environmental flakes, such as setupVirtualEnv seems
>> reasonable. I agree with Udi that care should be taken to not overload
>> Jenkins (e.g. retries should be limited)
>>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Alex Amato]

@Udi Meiri , Is this true if the specific tests are
rerun? I don't think we should rerun all tests.

On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev 
wrote:

> Preinstalling dependencies may affect the dependency resolution, and we
> may end up testing a different configuration than a user would have after
> installing beam into a clean environment.
>
> I do think pip uses cache, unless one specifies "--no-cache-dir". By
> default the cache is ~/.cache/pip. Looking up the log message in OP, we can
> see several "Using cached..." log entries. Not sure why futures was not
> fetched from cache or PyPi. Perhaps it is also a pip flake.
>
> I would be against wiping flakes under the rug by rerunning the whole
> suite after an error, but re-rerunning parts of the test environment set
> up, that are prone to environmental flakes, such as setupVirtualEnv seems
> reasonable. I agree with Udi that care should be taken to not overload
> Jenkins (e.g. retries should be limited)
>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Valentyn Tymofieiev]

Preinstalling dependencies may affect the dependency resolution, and we may
end up testing a different configuration than a user would have after
installing beam into a clean environment.

I do think pip uses cache, unless one specifies "--no-cache-dir". By
default the cache is ~/.cache/pip. Looking up the log message in OP, we can
see several "Using cached..." log entries. Not sure why futures was not
fetched from cache or PyPi. Perhaps it is also a pip flake.

I would be against wiping flakes under the rug by rerunning the whole suite
after an error, but re-rerunning parts of the test environment set up, that
are prone to environmental flakes, such as setupVirtualEnv seems
reasonable. I agree with Udi that care should be taken to not overload
Jenkins (e.g. retries should be limited)

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Pablo Estrada]

Good points, Kenn : ) - I know Yifan looked into running tests using
Docker. I imagine a testing container could have a set of dependencies
installed, and reduce the amount of installations we perform from PyPi.
Best
-P.

On Fri, Apr 26, 2019 at 11:49 AM Kenneth Knowles  wrote:

> Flakiness in Java got a lot better when we put the Maven cache outside the
> wiped build directory. I am not sure about Gradle now... It is obviously
> less hermetic, but these things should be immutable so a cache is
> acceptable.
>
> Is there a way to achieve this for Python? For Maven/Gradle a package
> being downloaded does not imply it is available for import. If downloaded
> == importable then you risk masking errors.
>
> Kenn
>
> On Fri, Apr 26, 2019 at 11:44 AM Udi Meiri  wrote:
>
>> That runs the risk of overloading our test machines when the build goes
>> red.
>>
>> On Fri, Apr 26, 2019 at 11:29 AM Alex Amato  wrote:
>>
>>> It would be ideal to not need manual steps. If known flakey tests can be
>>> auto retried that would be a great improvement.
>>>
>>> On Fri, Apr 26, 2019 at 11:24 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 We could do something along the lines of retry with a back-off. Note
 that Java tests also have this problem as we sometimes fail to fetch
 packages from Maven Central.

 On Fri, Apr 26, 2019 at 11:19 AM Pablo Estrada 
 wrote:

> hm no, these are somewhat common. Yes, I think we could have retries
> to try to fix this sort of problem.
>
> Perhaps a mixture of reusing a virtualenv, and having retries when
> creating it?
>
> On Fri, Apr 26, 2019 at 11:15 AM Alex Amato 
> wrote:
>
>> Okay but this occurred on jenkins. So does the machine need an update?
>>
>> On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> I think you hit a pypi flake.
>>>
>>> pip install futures>=2.2.0 works fine for me.
>>>
>>> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato 
>>> wrote:
>>>
 Would be nice to fix this as it can slow down PRs. I am not sure if 
 this one is fixed on retry yet or not.



 *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
 *



 *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
 *


 *18:05:44* >* Task 
 :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
 python executable in 
 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
  Also creating executable in 
 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* 
 Running virtualenv with interpreter /usr/bin/python2.7*18:05:44* 
 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 
 2020. Please upgrade your Python as Python 2.7 won't be maintained 
 after that date. A future version of pip will drop support for Python 
 2.7.*18:05:44* Collecting tox==3.0.0*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using 
 cached 
 https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using 
 cached 
 https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
 cached 
 

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Kenneth Knowles]

Flakiness in Java got a lot better when we put the Maven cache outside the
wiped build directory. I am not sure about Gradle now... It is obviously
less hermetic, but these things should be immutable so a cache is
acceptable.

Is there a way to achieve this for Python? For Maven/Gradle a package being
downloaded does not imply it is available for import. If downloaded ==
importable then you risk masking errors.

Kenn

On Fri, Apr 26, 2019 at 11:44 AM Udi Meiri  wrote:

> That runs the risk of overloading our test machines when the build goes
> red.
>
> On Fri, Apr 26, 2019 at 11:29 AM Alex Amato  wrote:
>
>> It would be ideal to not need manual steps. If known flakey tests can be
>> auto retried that would be a great improvement.
>>
>> On Fri, Apr 26, 2019 at 11:24 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> We could do something along the lines of retry with a back-off. Note
>>> that Java tests also have this problem as we sometimes fail to fetch
>>> packages from Maven Central.
>>>
>>> On Fri, Apr 26, 2019 at 11:19 AM Pablo Estrada 
>>> wrote:
>>>
 hm no, these are somewhat common. Yes, I think we could have retries to
 try to fix this sort of problem.

 Perhaps a mixture of reusing a virtualenv, and having retries when
 creating it?

 On Fri, Apr 26, 2019 at 11:15 AM Alex Amato  wrote:

> Okay but this occurred on jenkins. So does the machine need an update?
>
> On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> I think you hit a pypi flake.
>>
>> pip install futures>=2.2.0 works fine for me.
>>
>> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato 
>> wrote:
>>
>>> Would be nice to fix this as it can slow down PRs. I am not sure if 
>>> this one is fixed on retry yet or not.
>>>
>>>
>>>
>>> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
>>> *
>>>
>>>
>>>
>>> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
>>> *
>>>
>>>
>>> *18:05:44* >* Task 
>>> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
>>> python executable in 
>>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>>>  Also creating executable in 
>>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>>>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* 
>>> Running virtualenv with interpreter /usr/bin/python2.7*18:05:44* 
>>> DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 
>>> 2020. Please upgrade your Python as Python 2.7 won't be maintained 
>>> after that date. A future version of pip will drop support for Python 
>>> 2.7.*18:05:44* Collecting tox==3.0.0*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   
>>> Using cached 
>>> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Udi Meiri]

That runs the risk of overloading our test machines when the build goes red.

On Fri, Apr 26, 2019 at 11:29 AM Alex Amato  wrote:

> It would be ideal to not need manual steps. If known flakey tests can be
> auto retried that would be a great improvement.
>
> On Fri, Apr 26, 2019 at 11:24 AM Valentyn Tymofieiev 
> wrote:
>
>> We could do something along the lines of retry with a back-off. Note that
>> Java tests also have this problem as we sometimes fail to fetch packages
>> from Maven Central.
>>
>> On Fri, Apr 26, 2019 at 11:19 AM Pablo Estrada 
>> wrote:
>>
>>> hm no, these are somewhat common. Yes, I think we could have retries to
>>> try to fix this sort of problem.
>>>
>>> Perhaps a mixture of reusing a virtualenv, and having retries when
>>> creating it?
>>>
>>> On Fri, Apr 26, 2019 at 11:15 AM Alex Amato  wrote:
>>>
 Okay but this occurred on jenkins. So does the machine need an update?

 On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> I think you hit a pypi flake.
>
> pip install futures>=2.2.0 works fine for me.
>
> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:
>
>> Would be nice to fix this as it can slow down PRs. I am not sure if this 
>> one is fixed on retry yet or not.
>>
>>
>>
>> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
>> *
>>
>>
>>
>> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
>> *
>>
>>
>> *18:05:44* >* Task 
>> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
>> python executable in 
>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>>  Also creating executable in 
>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* 
>> Running virtualenv with interpreter /usr/bin/python2.7*18:05:44* 
>> DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 
>> 2020. Please upgrade your Python as Python 2.7 won't be maintained after 
>> that date. A future version of pip will drop support for Python 
>> 2.7.*18:05:44* Collecting tox==3.0.0*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using 
>> cached 
>> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using 
>> cached 
>> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>> cached 
>> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>> cached 
>> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>  Collecting futures>=2.2.0; python_version < "3.2" (from 
>> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
>> version that satisfies the requirement futures>=2.2.0; python_version < 
>> "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: 
>> none)*18:05:44* ERROR: No matching distribution found for 
>> futures>=2.2.0; python_version < "3.2" (from 
>> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* *18:05:46* >* Task 
>> :beam-sdks-python-test-suites-dataflow:setupVirtualenv* FAILED*18:05:46*
>>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Alex Amato]

It would be ideal to not need manual steps. If known flakey tests can be
auto retried that would be a great improvement.

On Fri, Apr 26, 2019 at 11:24 AM Valentyn Tymofieiev 
wrote:

> We could do something along the lines of retry with a back-off. Note that
> Java tests also have this problem as we sometimes fail to fetch packages
> from Maven Central.
>
> On Fri, Apr 26, 2019 at 11:19 AM Pablo Estrada  wrote:
>
>> hm no, these are somewhat common. Yes, I think we could have retries to
>> try to fix this sort of problem.
>>
>> Perhaps a mixture of reusing a virtualenv, and having retries when
>> creating it?
>>
>> On Fri, Apr 26, 2019 at 11:15 AM Alex Amato  wrote:
>>
>>> Okay but this occurred on jenkins. So does the machine need an update?
>>>
>>> On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 I think you hit a pypi flake.

 pip install futures>=2.2.0 works fine for me.

 On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:

> Would be nice to fix this as it can slow down PRs. I am not sure if this 
> one is fixed on retry yet or not.
>
>
>
> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
> *
>
>
>
> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
> *
>
>
> *18:05:44* >* Task 
> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
> python executable in 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>  Also creating executable in 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* 
> Running virtualenv with interpreter /usr/bin/python2.7*18:05:44* 
> DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 
> 2020. Please upgrade your Python as Python 2.7 won't be maintained after 
> that date. A future version of pip will drop support for Python 
> 2.7.*18:05:44* Collecting tox==3.0.0*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
> cached 
> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
> cached 
> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>  Collecting futures>=2.2.0; python_version < "3.2" (from 
> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
> version that satisfies the requirement futures>=2.2.0; python_version < 
> "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: 
> none)*18:05:44* ERROR: No matching distribution found for futures>=2.2.0; 
> python_version < "3.2" (from 
> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* *18:05:46* >* Task 
> :beam-sdks-python-test-suites-dataflow:setupVirtualenv* FAILED*18:05:46*
>
>  
> 
>
>
>
>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Valentyn Tymofieiev]

We could do something along the lines of retry with a back-off. Note that
Java tests also have this problem as we sometimes fail to fetch packages
from Maven Central.

On Fri, Apr 26, 2019 at 11:19 AM Pablo Estrada  wrote:

> hm no, these are somewhat common. Yes, I think we could have retries to
> try to fix this sort of problem.
>
> Perhaps a mixture of reusing a virtualenv, and having retries when
> creating it?
>
> On Fri, Apr 26, 2019 at 11:15 AM Alex Amato  wrote:
>
>> Okay but this occurred on jenkins. So does the machine need an update?
>>
>> On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> I think you hit a pypi flake.
>>>
>>> pip install futures>=2.2.0 works fine for me.
>>>
>>> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:
>>>
 Would be nice to fix this as it can slow down PRs. I am not sure if this 
 one is fixed on retry yet or not.



 *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
 *



 *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
 *


 *18:05:44* >* Task 
 :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
 python executable in 
 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
  Also creating executable in 
 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* 
 Running virtualenv with interpreter /usr/bin/python2.7*18:05:44* 
 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 
 2020. Please upgrade your Python as Python 2.7 won't be maintained after 
 that date. A future version of pip will drop support for Python 
 2.7.*18:05:44* Collecting tox==3.0.0*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
 https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
 cached 
 https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
 cached 
 https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
  Collecting futures>=2.2.0; python_version < "3.2" (from 
 grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
 version that satisfies the requirement futures>=2.2.0; python_version < 
 "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: 
 none)*18:05:44* ERROR: No matching distribution found for futures>=2.2.0; 
 python_version < "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* 
 *18:05:46* >* Task :beam-sdks-python-test-suites-dataflow:setupVirtualenv* 
 FAILED*18:05:46*

  
 

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Valentyn Tymofieiev]

No, just re-run the tests. For the record, there are PyPi error-rates in
https://status.python.org/#system-metrics.

On Fri, Apr 26, 2019 at 11:15 AM Alex Amato  wrote:

> Okay but this occurred on jenkins. So does the machine need an update?
>
> On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev 
> wrote:
>
>> I think you hit a pypi flake.
>>
>> pip install futures>=2.2.0 works fine for me.
>>
>> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:
>>
>>> Would be nice to fix this as it can slow down PRs. I am not sure if this 
>>> one is fixed on retry yet or not.
>>>
>>>
>>>
>>> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
>>> *
>>>
>>>
>>>
>>> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
>>> *
>>>
>>>
>>> *18:05:44* >* Task 
>>> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
>>> python executable in 
>>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>>>  Also creating executable in 
>>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>>>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* Running 
>>> virtualenv with interpreter /usr/bin/python2.7*18:05:44* DEPRECATION: 
>>> Python 2.7 will reach the end of its life on January 1st, 2020. Please 
>>> upgrade your Python as Python 2.7 won't be maintained after that date. A 
>>> future version of pip will drop support for Python 2.7.*18:05:44* 
>>> Collecting tox==3.0.0*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting futures>=2.2.0; python_version < "3.2" (from 
>>> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
>>> version that satisfies the requirement futures>=2.2.0; python_version < 
>>> "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: 
>>> none)*18:05:44* ERROR: No matching distribution found for futures>=2.2.0; 
>>> python_version < "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* 
>>> *18:05:46* >* Task :beam-sdks-python-test-suites-dataflow:setupVirtualenv* 
>>> FAILED*18:05:46*
>>>
>>>  
>>> 
>>>
>>>
>>>
>>>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Pablo Estrada]

hm no, these are somewhat common. Yes, I think we could have retries to try
to fix this sort of problem.

Perhaps a mixture of reusing a virtualenv, and having retries when creating
it?

On Fri, Apr 26, 2019 at 11:15 AM Alex Amato  wrote:

> Okay but this occurred on jenkins. So does the machine need an update?
>
> On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev 
> wrote:
>
>> I think you hit a pypi flake.
>>
>> pip install futures>=2.2.0 works fine for me.
>>
>> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:
>>
>>> Would be nice to fix this as it can slow down PRs. I am not sure if this 
>>> one is fixed on retry yet or not.
>>>
>>>
>>>
>>> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
>>> *
>>>
>>>
>>>
>>> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
>>> *
>>>
>>>
>>> *18:05:44* >* Task 
>>> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New 
>>> python executable in 
>>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>>>  Also creating executable in 
>>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>>>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* Running 
>>> virtualenv with interpreter /usr/bin/python2.7*18:05:44* DEPRECATION: 
>>> Python 2.7 will reach the end of its life on January 1st, 2020. Please 
>>> upgrade your Python as Python 2.7 won't be maintained after that date. A 
>>> future version of pip will drop support for Python 2.7.*18:05:44* 
>>> Collecting tox==3.0.0*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
>>> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>>>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>>> cached 
>>> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>>  Collecting futures>=2.2.0; python_version < "3.2" (from 
>>> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
>>> version that satisfies the requirement futures>=2.2.0; python_version < 
>>> "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: 
>>> none)*18:05:44* ERROR: No matching distribution found for futures>=2.2.0; 
>>> python_version < "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* 
>>> *18:05:46* >* Task :beam-sdks-python-test-suites-dataflow:setupVirtualenv* 
>>> FAILED*18:05:46*
>>>
>>>  
>>> 
>>>
>>>
>>>
>>>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Alex Amato]

Okay but this occurred on jenkins. So does the machine need an update?

On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev 
wrote:

> I think you hit a pypi flake.
>
> pip install futures>=2.2.0 works fine for me.
>
> On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:
>
>> Would be nice to fix this as it can slow down PRs. I am not sure if this one 
>> is fixed on retry yet or not.
>>
>>
>>
>> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
>> *
>>
>>
>>
>> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
>> *
>>
>>
>> *18:05:44* >* Task 
>> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New python 
>> executable in 
>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>>  Also creating executable in 
>> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* Running 
>> virtualenv with interpreter /usr/bin/python2.7*18:05:44* DEPRECATION: Python 
>> 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your 
>> Python as Python 2.7 won't be maintained after that date. A future version 
>> of pip will drop support for Python 2.7.*18:05:44* Collecting 
>> tox==3.0.0*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
>> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>> cached 
>> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
>> cached 
>> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>>  Collecting futures>=2.2.0; python_version < "3.2" (from 
>> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
>> version that satisfies the requirement futures>=2.2.0; python_version < 
>> "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: 
>> none)*18:05:44* ERROR: No matching distribution found for futures>=2.2.0; 
>> python_version < "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* 
>> *18:05:46* >* Task :beam-sdks-python-test-suites-dataflow:setupVirtualenv* 
>> FAILED*18:05:46*
>>
>>  
>> 
>>
>>
>>
>>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Valentyn Tymofieiev]

I think you hit a pypi flake.

pip install futures>=2.2.0 works fine for me.

On Fri, Apr 26, 2019 at 9:41 AM Alex Amato  wrote:

> Would be nice to fix this as it can slow down PRs. I am not sure if this one 
> is fixed on retry yet or not.
>
>
>
> *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 
> *
>
>
>
> *https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
> *
>
>
> *18:05:44* >* Task 
> :beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New python 
> executable in 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
>  Also creating executable in 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
>  Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44* Running 
> virtualenv with interpreter /usr/bin/python2.7*18:05:44* DEPRECATION: Python 
> 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your 
> Python as Python 2.7 won't be maintained after that date. A future version of 
> pip will drop support for Python 2.7.*18:05:44* Collecting 
> tox==3.0.0*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
>  Collecting grpcio-tools==1.3.5*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>  Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
>  Collecting six (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
>  Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
>  Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
>  Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using cached 
> https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>  Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*   Using 
> cached 
> https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
>  Collecting futures>=2.2.0; python_version < "3.2" (from 
> grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find a 
> version that satisfies the requirement futures>=2.2.0; python_version < "3.2" 
> (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from versions: none)*18:05:44* 
> ERROR: No matching distribution found for futures>=2.2.0; python_version < 
> "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* *18:05:46* >* Task 
> :beam-sdks-python-test-suites-dataflow:setupVirtualenv* FAILED*18:05:46*
>
>  
>
>
>
>

[BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

[Alex Amato]

Would be nice to fix this as it can slow down PRs. I am not sure if
this one is fixed on retry yet or not.



*https://issues.apache.org/jira/browse/BEAM-7164?filter=-2
*



*https://builds.apache.org/job/beam_PreCommit_Python_Commit/6035/consoleFull
*


*18:05:44* >* Task
:beam-sdks-python-test-suites-dataflow:setupVirtualenv**18:05:44* New
python executable in
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python2.7*18:05:44*
Also creating executable in
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-410805238/bin/python*18:05:44*
Installing setuptools, pkg_resources, pip, wheel...done.*18:05:44*
Running virtualenv with interpreter /usr/bin/python2.7*18:05:44*
DEPRECATION: Python 2.7 will reach the end of its life on January 1st,
2020. Please upgrade your Python as Python 2.7 won't be maintained
after that date. A future version of pip will drop support for Python
2.7.*18:05:44* Collecting tox==3.0.0*18:05:44*   Using cached
https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl*18:05:44*
Collecting grpcio-tools==1.3.5*18:05:44*   Using cached
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)*18:05:44*   Using
cached 
https://files.pythonhosted.org/packages/84/e8/4ddac125b5a0e84ea6ffc93cfccf1e7ee1924e88f53c64e98227f0af2a5f/pluggy-0.9.0-py2.py3-none-any.whl*18:05:44*
Collecting six (from tox==3.0.0)*18:05:44*   Using cached
https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl*18:05:44*
Collecting virtualenv>=1.11.2 (from tox==3.0.0)*18:05:44*   Using
cached 
https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl*18:05:44*
Collecting py>=1.4.17 (from tox==3.0.0)*18:05:44*   Using cached
https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl*18:05:44*
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)*18:05:44*   Using
cached 
https://files.pythonhosted.org/packages/7c/59/4da8df60a74f4af73ede9d92a75ca85c94bc2a109d5f67061496e8d496b2/grpcio-1.20.0-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)*18:05:44*
Using cached 
https://files.pythonhosted.org/packages/ea/72/5eadea03b06ca1320be2433ef2236155da17806b700efc92677ee99ae119/protobuf-3.7.1-cp27-cp27mu-manylinux1_x86_64.whl*18:05:44*
Collecting futures>=2.2.0; python_version < "3.2" (from
grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:44*   ERROR: Could not find
a version that satisfies the requirement futures>=2.2.0;
python_version < "3.2" (from grpcio>=1.3.5->grpcio-tools==1.3.5) (from
versions: none)*18:05:44* ERROR: No matching distribution found for
futures>=2.2.0; python_version < "3.2" (from
grpcio>=1.3.5->grpcio-tools==1.3.5)*18:05:46* *18:05:46* >* Task
:beam-sdks-python-test-suites-dataflow:setupVirtualenv*
FAILED*18:05:46*