On Fri, Feb 28, 2020 at 6:09 AM Wols Lists <antli...@youngman.org.uk> wrote: > > On 27/02/20 21:49, Rich Freeman wrote: > > A fairly cheap amd64 system can run a ton of services in containers > > though, and it is way simpler to maintain that way. I still get quick > > access to snapshots/etc, but now if I want to run a gentoo container > > it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one > > core, but once a month it needs 4GB of RAM and 100% of 6 cores. As > > long as I'm not doing an emerge -u world on half a dozen containers at > > once it is no big deal at all. > > Do all your containers have the same make options etc? Can't remember > which directory it is, but I had a shared emerge directory where it > stored this stuff and I emerged with -bk options (use binary if it's > there, create binary if it isn't). >
They're probably not too far off in general, but not exact. I only run one instance of any particular container, so I haven't tried to do parallel builds. If portage had support for multiple binary packages co-existing with different build options I might. If I ever get really bored for a few weeks I could see playing around with that. It seems like it ought to be possible to content-hash the list of build options and stick that hash in the binary package filename, and then have portage search for suitable packages, using a binary package if one matches, and doing a new build if not. Many of my containers don't even run Gentoo. I have a few running Arch, Ubuntu Server, or Debian. If some service is well-supported in one of those and is poorly supported in Gentoo I will tend to go that route. I'll package it if reasonable but some upstreams are just not very conducive to this. There was a question about ARM-based NAS in this thread which I'll go ahead and tackle to save a reply. I'm actually playing around with lizardfs (I might consider moosefs instead if starting from scratch - or Ceph if I were scaling up but that wouldn't be practical on ARM). I have a mix of chunkservers but my target is to run new ones on ARM. I'm using RockPro64 SBCs with LSI HBAs (this SBC is fairly unique in having PCIe). There is some issue with the lizardfs code that causes performance issues on ARM though I understand they're working on this, so that could change. I'm using it for multimedia and I care more about static space than iops, so it is fine for me. The LSI HBA pulls more power than the SBC does, but overall the setup is very low-power and fairly inexpensive (used HBAs on ebay). I can in theory get up to 16 drives on one SBC this way. The SBC also supports USB3 so that is another option with a hub - in fact I'm mostly shucking USB3 drives anyway. Main issue with ARM SBCs in general is that they don't have much RAM, so IMO that makes Ceph a non-starter. Otherwise that would probably be my preferred option. Bad things can happen on rebuilds if you don't have 1GB/TB as they suggest, and even with the relatively under-utilized servers I have now that would be a LOT of RAM for ARM (really, it would be expensive even on amd64). Lizardfs/moosefs chunkservers barely use any RAM at all. The master server does need more - I have shadow masters running on the SBCs but since I'm using this for multimedia the metadata server only uses about 100MB of RAM and that includes processes, libraries, and random minimal service daemons like sshd. I'm running my master on amd64 though to get optimal performance, shadowed on the chunkservers so that I can failover if needed, though in truth the amd64 box with ECC is the least likely thing to die and runs all the stuff that uses the storage right now anyway. The other suggestion to consider USB3 instead of SATA for storage isn't a bad idea. Though going that route means wall warts and drives as far as the eye can see. Might still be less messy than my setup, which has a couple of cheap ATX PSUs with ATX power switches, 16x PCIe powered risers for the HBAs (they pull too much power for the SBC), and rockwell drive cages to stack the drives in (they're meant for a server chasis but they're reasonably priced and basically give you an open enclosure with a fan). I'd definitely have a lot fewer PCBs showing if I used USB3 instead. I'm not sure how well that would perform though - that HBA has a lot of bandwidth if the node got busy with PCIe v2 x4 connectivity (SAS9200-16E) and with USB3 it would all go through 1-2 ports. Though I doubt I'd ever get THAT many drives on a node and if I needed more space I'd probably expand up to 5 chunkservers before I'm putting more than about 3 drives on each - you get better performance and more fault-tolerance that way. One big reason I went the distributed filesystem approach was that I was getting tired of trying to cram as many drives as I could into a single host and then dealing with some of the inflexibilities of zfs. The inflexibility bit is improving somewhat with removable vdevs, though I'm not sure how much residue those leave behind if you do it often. But, zfs is still limited to however many drives you can cram into one host, while a distributed filesystem lets you expand outwards. Plus it is fault-tolerant at the host level instead of the drive level. -- Rich
