On Wed, Nov 29, 2017 at 9:09 PM, Thanh Ha <[email protected]> wrote:
> Yes, that's right the systems have 4GB of RAM now. If the only things we > need swap for is the robot systems then I'm more inclined to suggest we > create it as part of the job definition first, and move it to packer if we > find it adds to much build time (I'm not sure if creating a swap file is a > significant time process, it's been a long time since I had to make one). > could also create robot vm flavors based on the ram. You really only need the 4gb robot when processing the larger log.html - which netvirt just started hitting last few weeks. The output.xml is going up to 1gb in size. I don't think any other projects hit that. I guess right now all the jobs are using the 4gb robot - but no other project was having issues. So this might be the easiest thing to just switch netvirt to using the larger robot vm. agreed on swap is slow ram - that suggestion was only if increasing ram wasn't an option. > > Cheers, > Thanh > > > > On Wed, Nov 29, 2017 at 1:22 PM, Jamo Luhrsen <[email protected]> wrote: > >> actually, I get your point, Luis. There is always a little give and take >> when we >> want to ask for more RAM. I take it that swap space is not going to be a >> big deal >> since storage is not something we are every worried about. >> >> right Thanh? >> >> >> btw, we did bump the robot builder vms from 2G to 4G RAM recently. >> >> JamO >> >> On 11/29/2017 09:57 AM, Luis Gomez wrote: >> > OK, if we have RAM lets add it, I thought we had some constraint there. >> > >> >> On Nov 29, 2017, at 9:43 AM, Jamo Luhrsen <[email protected] <mailto: >> [email protected]>> wrote: >> >> >> >> Well, I think Thanh's point is that if the output.xml is too big, we >> just add more RAM to >> >> fix it, although instead we could add some swap which is "slower RAM" >> >> >> >> JamO >> >> >> >> On 11/29/2017 09:13 AM, Luis Gomez wrote: >> >>> I do not remember all the issues related with memory but at least >> swap is required in the robot VM to generate the log.html >> >>> when output.xml file is big. >> >>> >> >>>> On Nov 29, 2017, at 2:56 AM, Thanh Ha <[email protected] >> >>>> <mailto:[email protected]> <mailto:thanh.ha@ >> linuxfoundation.org>> wrote: >> >>>> >> >>>> We can certainly add swap but imo swap is just a slower ram. At >> least in my experience when a system starts needing to swap >> >>>> things have already gone beyond recovery and swap just delays the >> inevitable program crash. In any case there's a few ways >> >>>> we can add swap if we really want to: >> >>>> >> >>>> 1) At the job level, create a swap file and activate it for the size >> needed for the job. >> >>>> 2) At the jenkins-init level (via jenkins-scripts in releng/builder) >> we can add swap for systems here >> >>>> 3) Bake it into the VM via packer file >> >>>> >> >>>> If we want to try this with a specific job type then I would try it >> with option 1 first and see how things go. If we find >> >>>> it really useful we can move it into 2 or 3. >> >>>> >> >>>> Regards, >> >>>> Thanh >> >>>> >> >>>> On Mon, Nov 13, 2017 at 6:19 PM, Luis Gomez <[email protected] >> <mailto:[email protected]> <mailto:[email protected]>> wrote: >> >>>> >> >>>> +1, adding Andy: is it possible to get swap space in the VM from >> the cloud provider? otherwise is it possible to modify >> >>>> the releng/builder packer scripts to add the swap? >> >>>> >> >>>>> On Nov 13, 2017, at 3:05 PM, Sam Hague <[email protected] >> <mailto:[email protected]> <mailto:[email protected]>> wrote: >> >>>>> >> >>>>> On Thu, Nov 9, 2017 at 5:45 AM, Stephen Kitt <[email protected] >> <mailto:[email protected]> <mailto:[email protected]>> >> >>>>> wrote: >> >>>>> >> >>>>> Exactly, that’s what I was alluding to with “the lack of >> swap probably >> >>>>> doesn’t help” ;-). We should really configure VMs with a >> small amount >> >>>>> of swap, it can save the kernel in tricky situations... >> >>>>> >> >>>>> I was looking at this also and wondering about swap. We had a >> different issue where the robot VMs blow up on producing >> >>>>> the log.html file because it ends up being a 1gb file. With swap >> I think it would have passed fine. As is, we bumped >> >>>>> the 2gb vm to a 4gb to make it work. So we will likely hit it >> again. >> >>>>> >> >>>>> >> >>>>> On Thu, 9 Nov 2017 01:47:08 -0800 >> >>>>> Anil Vishnoi <[email protected] <mailto: >> [email protected]> <mailto:[email protected]>> wrote: >> >>>>> >> >>>>>> I suspect you might hit it again if you run it a bit longer because >> >>>>>> of this >> >>>>>> >> >>>>>> [Thu Nov 2 05:58:08 2017] Free swap = 0kB >> >>>>>> [Thu Nov 2 05:58:08 2017] Total swap = 0kB >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Nov 9, 2017 at 1:39 AM, Stephen Kitt <[email protected] >> <mailto:[email protected]> <mailto:[email protected]>> wrote: >> >>>>>> >> >>>>>>> On Thu, 9 Nov 2017 10:28:14 +0100 >> >>>>>>> Robert Varga <[email protected] <mailto:[email protected]> <mailto:[email protected]>> >> wrote: >> >>>>>>>> On 02/11/17 23:02, Luis Gomez wrote: >> >>>>>>>>> 1) JVM does not kill itself, the OS does instead after the java >> >>>>>>>>> process grows to 3.7G in a VM of 4G RAM (note Xmx is set to 2G >> >>>>>>>>> but still the jvm goes far beyond that). >> >>>>>>>> >> >>>>>>>> Indicates this lies out side of heap -- check thread count. >> >>>>>>> >> >>>>>>> We verified separately that this is an OOM issue, but one detected >> >>>>>>> by the kernel rather than by the JVM (the OOM killer kills the >> JVM, >> >>>>>>> see >> >>>>>>> https://jira.opendaylight.org/secure/attachment/14207/dmesg. >> log.txt >> >>>>> <https://jira.opendaylight.org/secure/attachment/ >> 14207/dmesg.log.txt> >> >>>>>>> for details; the number of threads wasn’t an issue here, but the >> >>>>>>> lack of swap probably didn’t help). >> >>>>>>> >> >>>>>>> Upgrading to OpenJDK 8 patch 151 fixed the problem, it might have >> >>>>>>> been related to one of the several memory usage bugs in 144 that >> >>>>>>> were fixed in 151. It’s probably just moving the goalposts though >> >>>>>>> since the problem was new — basically, I suspect we recently >> >>>>>>> started using a little too much off-heap memory for some reason, >> >>>>>>> and the upgrade to 151 reduces the JVM’s memory usage enough to >> >>>>>>> make us fit in our VMs again. >> >>>>>>> >> >>>>>>> Regards, >> >>>>>>> >> >>>>>>> Stephen >> >>>> >> >>>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> integration-dev mailing list >> >>> [email protected] <mailto:integration-dev@lists. >> opendaylight.org> >> >>> https://lists.opendaylight.org/mailman/listinfo/integration-dev >> > >> > > > _______________________________________________ > integration-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/integration-dev > >
_______________________________________________ controller-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/controller-dev
