David, FYI - it will impact VirtualBox trials. 3 controllers x 3GB = 9 GB RAM only for controllers if you run HA.
Roman On Thu, Aug 21, 2014 at 5:22 AM, Sergii Golovatiuk <sgolovat...@mirantis.com > wrote: > +1 for not using debug at CI gates (only BVT) > +1 for adding atop to our builds as it helps to understand what was wrong > at that particular time > +1 for increasing RAM (though we'll try to tune rabbit and Galera). 3GB > should be enough so we'll be able to run up to 2 environments on 32GB RAM > servers. > > -- > Best regards, > Sergii Golovatiuk, > Skype #golserge > IRC #holser > > > On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <bdobre...@mirantis.com> > wrote: > >> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote: >> > Hi, >> > >> > Digging the issue with Galera, I found that our environments have very >> > high RAM utilization which leads to the problem during environment >> > deployment. For instance "HA deployment + neutron/GRE" requires almost >> > 2.6-2.7 GB during deployment >> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I found >> > high swap in/swap out usage during deployment with very high load >> > average. This creates many sporadic issues with some services. They time >> > out in random place making our debugging very hard. I would like to >> > review our policy for CI environment and increase RAM (at least for bvt >> > tests) to 3 GB. >> > >> > -- >> > Best regards, >> > Sergii Golovatiuk, >> > Skype #golserge >> > IRC #holser >> > >> > >> >> I believe we should do at least the following for our CI jobs and bvt >> tests: >> 1) Deployment shortcuts: stop deployment abruptly, if any deployment >> blocker has been met, such as something exceeded given # of retries. >> That could be done in puppet by overriding 'tries' behavior in exec >> provider, or at orchestration layer as well. >> 2) Load management: collect and automatically analyze atop stats (swap >> rates, load average, io waiters) from jenkins slaves and vm nodes while >> running the jobs, and stop or freeze some jobs, if some >> performance-stopper criteria has been met as well. >> 3) Do not use debug level logging for CI gates, use it only for bvt tests. >> >> -- >> Best regards, >> Bogdan Dobrelya, >> Skype #bogdando_at_yahoo.com >> Irc #bogdando >> > > > -- > Mailing list: https://launchpad.net/~fuel-dev > Post to : fuel-dev@lists.launchpad.net > Unsubscribe : https://launchpad.net/~fuel-dev > More help : https://help.launchpad.net/ListHelp > >
-- Mailing list: https://launchpad.net/~fuel-dev Post to : fuel-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
