+1 for not using debug at CI gates (only BVT) +1 for adding atop to our builds as it helps to understand what was wrong at that particular time +1 for increasing RAM (though we'll try to tune rabbit and Galera). 3GB should be enough so we'll be able to run up to 2 environments on 32GB RAM servers.
-- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <bdobre...@mirantis.com> wrote: > On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote: > > Hi, > > > > Digging the issue with Galera, I found that our environments have very > > high RAM utilization which leads to the problem during environment > > deployment. For instance "HA deployment + neutron/GRE" requires almost > > 2.6-2.7 GB during deployment > > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I found > > high swap in/swap out usage during deployment with very high load > > average. This creates many sporadic issues with some services. They time > > out in random place making our debugging very hard. I would like to > > review our policy for CI environment and increase RAM (at least for bvt > > tests) to 3 GB. > > > > -- > > Best regards, > > Sergii Golovatiuk, > > Skype #golserge > > IRC #holser > > > > > > I believe we should do at least the following for our CI jobs and bvt > tests: > 1) Deployment shortcuts: stop deployment abruptly, if any deployment > blocker has been met, such as something exceeded given # of retries. > That could be done in puppet by overriding 'tries' behavior in exec > provider, or at orchestration layer as well. > 2) Load management: collect and automatically analyze atop stats (swap > rates, load average, io waiters) from jenkins slaves and vm nodes while > running the jobs, and stop or freeze some jobs, if some > performance-stopper criteria has been met as well. > 3) Do not use debug level logging for CI gates, use it only for bvt tests. > > -- > Best regards, > Bogdan Dobrelya, > Skype #bogdando_at_yahoo.com > Irc #bogdando >
-- Mailing list: https://launchpad.net/~fuel-dev Post to : fuel-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
