On Sun, Dec 4, 2016 at 9:59 AM, Barak Korren <bkor...@redhat.com> wrote: > On 3 December 2016 at 21:36, Nir Soffer <nsof...@redhat.com> wrote: >> HI all, >> >> Watching vdsm travis builds in the last weeks, it is clear that vdsm tests >> on travis are about 4X times faster compared with jenkins builds. >> >> Here is a typical build: >> >> ovirt ci: >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/consoleFull >> travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079 >> >> The build took 4:34 on travis, and 19:34 on ovirt ci. > > Interesting, thanks for looking at this! > >> >> This has a huge impact on vdsm maintainers. Having to wait 20 minutes >> for each patch >> means that we must ignore the ci and merge and hope that previous tests >> without >> rebasing on master were good enough. >> >> The builds are mostly the same, expect: >> >> - In travis we don't check if the build system was changed and >> packages should be built >> takes 9:18 minutes in ovirt ci. > > Well, I guess the infra team can't help with that, but still, is there > anything we could do at the infrastructure level to speed this up?
The line taking 9 minutes is: if git diff-tree --no-commit-id --name-only -r HEAD | egrep --quiet 'vdsm.spec.in|Makefile.am' ; then ./automation/build-artifacts.sh yum -y install "$EXPORT_DIR/"!(*.src).rpm fi In the build above the condition is false, and we not run build-artifacts.sh or installing the packages. The time is spent in: git diff-tree --no-commit-id --name-only -r HEAD | egrep --quiet 'vdsm.spec.in|Makefile.am' Running locally: $ time git diff-tree --no-commit-id --name-only -r HEAD | egrep --quiet 'vdsm.spec.in|Makefile.am' real 0m0.009s user 0m0.006s sys 0m0.009s To debug this we need to get a shell on a jenkins slave with the exact environment of a running job. > >> - In travis we don't clean or install anything before the test, we use >> a container with all the >> available packages, pulled from dockerhub. >> takes about 3:52 minutes in ovirt ci > > Well, I guess this is where we (the infra team) should pay attention. > We do have a plan to switch from mock to Docker at some point > (OVIRT-873 [1]), but it'll take a while until we can make such a large > switch. > > It the meantime there may be some low-hanging fruit we can pick to > make things faster. Looking at the same log: > > 16:03:28 Init took 77 seconds > > 16:05:50 Install packages took 142 seconds > > We may be able to speed those up - looking at the way muck is > configured, we may be running with its caches turned off (I'm not yet > 100% sure about this - muck_runner.sh is not the simplest script...). > I've created OVIRT-902 [2] for us to look at this. > >> - In travis we don't enable coverage. Running the tests with coverage >> may slow down the tests >> takes 5:04 minutes in ovirt ci >> creating the coverage report takes only 15 seconds, not interesting > > We can easily check this by just sending a patch with coverage turned > on and then sending another patch set for the same patch with coverage > turned off. > >> - In travis we don't cleanup anything after the test >> this takes 34 seconds in ovirt ci > > We can look at speeding this up - or perhaps just change things so > that results are reported as soon as check_patch.sh is done as opposed > to when the Jenkins job is done. > There may be some pitfalls here so I need to think a little more > before I recommend going down this path. > >> The biggest problem is the build system check taking 9:18 minutes. >> fixing it will cut the build time in half. > > Please try fixing that, or maybe this should just move to build_artifacts.sh? > > [1]: https://ovirt-jira.atlassian.net/browse/OVIRT-873 > [2]: https://ovirt-jira.atlassian.net/browse/OVIRT-902 > > -- > Barak Korren > bkor...@redhat.com > RHCE, RHCi, RHV-DevOps Team > https://ifireball.wordpress.com/ _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel