FYI - Here is a POC build of this concept running. This only runs the Parser and Enrichment integration tests, but as separate jobs in Travis.
https://travis-ci.org/nickwallen/metron/builds/535791047 On Wed, May 22, 2019 at 9:20 AM Nick Allen <n...@nickallen.org> wrote: > > > Justin Leet said > <https://github.com/apache/metron/pull/1417#issuecomment-494464795> [1]: > Looking > at our build times, I'm actually concerned that if we kill the caches our > builds won't complete. The integration tests take >45 minutes and it's very > possible redownloading everything goes over our remaining time. > > To recap, our current Travis CI build is composed of multiple jobs. Travis' > time limit of 50 minutes > <https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts> [2] > is per job, rather than the total build time. While our total build time > is on the order of 2.5 hours, it is only our integration-test job which is > coming close to that 50 minute limit. > > We could try splitting our integration tests into multiple jobs. Each of > these would have the opportunity to run in parallel (given whatever > resources Travis can allocate to us), but more importantly each one has its > own 50 minute limit. For example... > > - Parser Integration Tests: > - `time mvn surefire:test@integration-tests -pl > "metron-platform/metron-parsing/metron-parsing-storm, > metron-platform/metron-parsing/metron-parsers, > metron-platform/metron-parsing/metron-parsers-common/"` > - Enrichment Integration Tests > - `time mvn surefire:test@integration-tests -pl > > "metron-platform/metron-enrichment/metron-enrichment-common/,metron-platform/metron-enrichment/metron-enrichment-storm"` > - etc, etc > > > We would just need to determine how to logically split the tests up. If > this sounds reasonable, I've already got a start on a POC. > > --- > [1] https://github.com/apache/metron/pull/1417#issuecomment-494464795 > [2] https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts >
