On Mon, 17 Oct 2016 13:35:47 +0100 Tom Hacohen <t...@osg.samsung.com> said:
> On 09/10/16 10:15, Jonathan Aquilina wrote: > > Hi Carsten, > > > > I think a good way forward would be to get a small server, I would even > > be willing to sponsor a vps and use puppet. It would make automation > > deployments and system management alot easier. I am going to be setting > > up puppet for my own systems. Would you be interested in such > > automation? Also what about infrastructure monitoring? > > Just a side comment that has only been casually mentioned. > > Beber is a bonafide sysadmin. He's been maintaining our infra for a long > time. We have a complex infra already in store. I think he uses "salt" > (a puppet alternative) to manage our numerous servers, though I'm not > sure. Help would always be appreciated, but it should go through him in > order to avoid trouble. that's why i've deferred to beber and jon talking things over. this seems to have not happened in the past weeks. i think beber is just really busy with other stuff. he is indeed a professional admin and a good guy. i think what is needed is just some more eyeballs, poking around and watching to see what might be causing this. > The problem here is not with server X, or server Y. The problem here is > with the host node hosting our VMs. It is the one having issues. Maybe > as raster implied later in this thread, it just needs to be updated. Who > knows. my current guess is possibly a kernel issue. the machine still runs 3.18 and i've noticed some kvm: zapping shadow pages for mmio generation wraparound dmesg logs that happen when things go bad, but i dont remember seeing them before when things are working. i am wondering if these are related. my googling says the message is harmless but i am not so sure. perhaps newer kernels have fixed this. i actually started doing a kernel upgrade (using genkerenl) but the kernel simply doesnt compile (missing symbols) ... and at this point i go "wtf? gentoo? kernel not compiling? AAAAARGH". and i find other things on my todo list as the time i then have to sink into figuring out the issue is less than the time to just reboot the machine next time it hiccups. also given the completely manual nature of kernels on gnetoo it appears i'm really hesitant even if a new kernel builds, to use it as if i forget something and things go wrong... i can't readily fix it due to no console access. :( > -- > Tom. > > > > > On 2016-10-09 11:10, Carsten Haitzler wrote: > > > >> On Sun, 09 Oct 2016 10:57:57 +0200 Jonathan Aquilina > >> <jaquil...@eagleeyet.net> said: > >> > >>> what can i do to help get things sorted out as you had mentioned below > >>> though. I am eager to help. > >> > >> well talk with beber. if he trusts you then some debugging. the system is > >> running an oldish 3.18 kernel so who knows if an update fixes it or > >> not. :) we could so with cutting down our storage usage as we use 750g or > >> so right now out of 1tb. i think beber could do with help with some > >> deployment work - andy williams asked a few weeks back to set u a vm for a > >> project of his. > >> > >> On 2016-10-09 10:55, Carsten Haitzler wrote: > >> > >> On Sun, 09 Oct 2016 08:17:23 +0200 Jonathan Aquilina > >> <jaquil...@eagleeyet.net> said: > >> > >> Regarding version control repos, why not offload to github which is free > >> even for those projects which are open source. > >> hell no. after having offloaded to sourecforge and finding servers going > >> down or things needing to be done that require admin control, no. we run > >> ours because anything else is generally worse. > >> > >> Bertrand, where do you need a hand. If you are saying this is a kernel > >> issue why not update the system and reboot. Also what is the host system > >> running distro wise? > >> > >> On 2016-10-09 02:20, Carsten Haitzler wrote: > >> > >> On Sat, 08 Oct 2016 11:16:28 +0200 Jonathan Aquilina > >> <jaquil...@eagleeyet.net> said: > >> > >> anything I can help with. Would a server from soyoustart.com help in > >> anyway. I am willing to get the project one and take charge of its > >> maintenance > >> we have a very big beefy server. we're not running out of disk space, > >> processing power etc. (8 core xeon @ 2.2ghz, 48gb ram, 1tb or so of disk > >> space available across both ssd and hdd fully redundant raid0'd). > >> > >> the issue is a software issue i suspect in the host system kernel and we > >> just don't know why or what. (my suspicious is due to it needing a reboot > >> to fix). > >> > >> we have used a fair bit of disk space but tbh the largest user is jenkins > >> (253gb) and it probably could do with a cleanup after a few years to lean > >> it back down. we probably could shut down and archive the cvs and svn vm's > >> )extract out the cvs and svn db data to a compressed tarball and make that > >> available and otherwise nuke the vm's). that should cut that down to 1-2gb > >> saving about 18gb. web is using 52gb - i think that might do with some > >> trimming/compacting. phab is using 55gb - same. playschool is 66gb - maybe > >> trim too and re-compact down? e5v1 is pretty fat too (29g). the build vm's > >> are pretty bug (10 of them ranging from 270k to 65g - funny the windows10 > >> vm for builds is only 18g. the gentoo ones are mostly 60gb or so - windows > >> is leaner than gentoo! :) )... and of course jenkins is at 253g. > >> > >> actually man we're fat on disk space usage. we really could do with a trim > >> there. the images are qcow2 images. we need to trim. > >> > >> On 2016-10-07 13:42, Mike Blumenkrantz wrote: > >> > >> Is there some way we could prevent this from happening? All the services > >> were unavailable for 8+ hours yesterday and it was incredibly disruptive. > >> > >> On Thu, Oct 6, 2016 at 11:09 PM Bertrand Jacquin <bertr...@jacquin.bzh> > >> wrote: > >> > >> Unspanked(tm) > >> > >> On Fri, Oct 07, 2016 at 12:18:07PM +1030, Simon Lees wrote: *** SPANK SPANK > >> SPANK!!! Enlightenment server is over capacity > >> > >> Please wait a moment and try again later. > >> For more information, take a look at #e on IRC, server irc.freenode.org. > >> > >> -- > >> > >> Simon Lees (Simotek) http://simotek.net > >> > >> Emergency Update Team keybase.io/simotek > >> SUSE Linux Adeliade Australia, UTC+9:30 > >> GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B > >> > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most engaging > >> tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ enlightenment-devel mailing > >> list enlightenment-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > >> > >> -- > >> Bertrand > >> > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> enlightenment-devel mailing list > >> enlightenment-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> enlightenment-devel mailing list > >> enlightenment-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> enlightenment-devel mailing list > >> enlightenment-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > enlightenment-devel mailing list > > enlightenment-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > enlightenment-devel mailing list > > enlightenment-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > enlightenment-devel mailing list > > enlightenment-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > _______________________________________________ > enlightenment-devel mailing list > enlightenment-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) ras...@rasterman.com ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
