It is possible to download dependencies with pip to a local directory and install from there [1]. As a side benefit this is supposed to speed up the installation process. Since we setup virtualenv multiple times, this could actually help us in a single run. And if we can keep this cache across test runs we can reduce flakiness.
[1] https://pip.pypa.io/en/latest/user_guide/#installing-from-local-packages On Fri, Apr 26, 2019 at 3:42 PM Valentyn Tymofieiev <[email protected]> wrote: > We do retry certain inherently flaky tests, for example, see[1]. This > practice should be used with caution, see discussion [2]. > > However retrying an individual test would not avoid the flake that Alex > brought up in this thread, we'd have to retry setupVirtualEnv task that is > executed once per suite of tests. Retrying just that task is different from > retrying the whole suite. > > [1] > https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41, > this was discussed > [2] > https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E > discussed > > On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri <[email protected]> wrote: > >> Alex, I changed my mind: I'm okay retrying single tests, just not entire >> suites of tests (e.g. if precommits take an hour, retrying the run takes up >> an additional hour on the Jenkins machine). >> This is more of an issue in Python, where gradle does not (currently) >> have insight into which tests failed and how to retry just them. >> >> >> >> On Fri, Apr 26, 2019 at 2:17 PM Alex Amato <[email protected]> wrote: >> >>> @Udi Meiri <[email protected]>, Is this true if the specific tests are >>> rerun? I don't think we should rerun all tests. >>> >>> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev < >>> [email protected]> wrote: >>> >>>> Preinstalling dependencies may affect the dependency resolution, and we >>>> may end up testing a different configuration than a user would have after >>>> installing beam into a clean environment. >>>> >>>> I do think pip uses cache, unless one specifies "--no-cache-dir". By >>>> default the cache is ~/.cache/pip. Looking up the log message in OP, we can >>>> see several "Using cached..." log entries. Not sure why futures was not >>>> fetched from cache or PyPi. Perhaps it is also a pip flake. >>>> >>>> I would be against wiping flakes under the rug by rerunning the whole >>>> suite after an error, but re-rerunning parts of the test environment set >>>> up, that are prone to environmental flakes, such as setupVirtualEnv seems >>>> reasonable. I agree with Udi that care should be taken to not overload >>>> Jenkins (e.g. retries should be limited) >>>> >>>
