- How many resources to ValidatesRunner tests use? - Where are those settings?
On Sat, Jun 30, 2018 at 9:50 AM Reuven Lax <re...@google.com> wrote: > The specific issue only affects Dataflow ValidatesRunner tests. We > currently allow only one of these to run at a time, to control usage of > Dataflow and of GCE quota. Other types of tests do not suffer from this > issue. > > I would like to see if it's possible to increase Dataflow quota so we can > run more of these in parallel. It took me 8 hours end to end to run these > tests (about 6 hours for the run to be scheduled). If there was a failure, > I would have had to repeat the whole process. In the worst case, this > process could have taken me days. While this is not as pressing as some > other issues (as most people don't need to run the Dataflow tests on every > PR), fixing it would make such changes much easier to manage. > > Reuven > > On Sat, Jun 30, 2018 at 9:32 AM Rafael Fernandez <rfern...@google.com> > wrote: > >> +Reuven Lax <re...@google.com> told me yesterday that he was waiting for >> some test to be scheduled and run, and it took 6 hours or so. I would like >> to help reduce these wait times by increasing parallelism. I need help >> understanding the continuous minimum of what we use. It seems the following >> is true: >> >> >> - There seems to always be 16 jenkins machines on (16 CPUs each) >> - There seems to be three GKE machines always on (1 CPU each) >> - Most (if not all) unit tests run on 1 machine, and seem to run >> one-at-a-time <-- I think we can safely parallelize this to 20. >> >> With current quotas, if we parallelize to 20 concurrent unit tests, we >> still have room for 80 other concurrent dataflow jobs to execute, with 75% >> of CPU capacity. >> >> Thoughts? Additional data? >> >> Thanks, >> r >> >
smime.p7s
Description: S/MIME Cryptographic Signature
