| hashar added a comment. |
In T162166#3158359, @chasemp wrote:Using swap seems like a non-starter unless we are totally devoid of RAM capacity in which case we have a lot more problems. I don't want to start conflating IO load with RAM issues on CI and it will only hurt the tests themselves. For the record we do not overallocate RAM as it has caused nothing but issues in the past.
Why did this start happening? What changed? (Best I can find is https://phabricator.wikimedia.org/T125050#3140483 which seems recursive)
As code and tests are being added, we eventually approached a limit when there is sometime not enough memory to allow for the fork() to happens. Adding swap solve that issue since Linux would have a guarantee that the forked process would eventually be able to allocate the whole copy on write memory.
Definitely agree that in case swap ends up being (and surely it will) the IO would be concerning. So I guess we can dismiss the idea of adding swap entirely.
Do we need 40G of / for these instances and why?
That has been set arbitrarily, usage on idling instances is 3G / 41G for Trusty and 4.8G / 41G for Jessie. So I guess we can go from 40G to 10G and we will be fine.
Do we have the ability to target certain flavors at certain tests? (as it seems a bit naive on our part to say that the only available instance size is the one big enough to run our biggest test which will also be used to run our smallest)
Yes that is definitely doable in Nodepool.
We would create a new snapshot image that uses the same disk image (image-ci-trusty) but require a different flavor that has more RAM. Then in Nodepool define a new Jenkins label attached to that new instance with some min-ready, and migrate the Jenkins job to that new label.
Cc: chasemp, StudiesWorld, JanZerebecki, Aklapper, hoo, Paladox, thcipriani, aude, Anomie, WMDE-leszek, Ladsgroup, Aleksey_WMDE, Lydia_Pintscher, Stashbot, Legoktm, greg, hashar, QZanden, Tbscho, Salgo60, D3r1ck01, Izno, Wikidata-bugs, Dinoguy1000, Gryllida, jayvdb, MrStradivarius, scfc, Jackmcbarn, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
