Hello Ian, Michael, Bob, Corwin, Amin, First of all, my condolences to Ian's and Michael's livers. No one should have to work an all-nighter, so that's why I'm here hoping I can help.
As I proposed on IRC, I can offer my VMs (& my expertise) to FSF so that
hopefully they can provide fail-over or load-balancing. I have a cluster of
servers that consists of various different spec, some are freedom-friendly and
some are less so. So here is a detailed list of machines I own that can host
VMs and each of their pros and cons (in my opinion) for your consideration
(also for future documenting purpose). Please let me know what you think and
what they can provide for FSF (e.g. DNS, MariaDB cluster, web etc.).
Suggestions are very welcome.
Geolocation: Tokyo, Japan
Electricity: 100% renewable energy.
Internet (not a machine but worth mentioning):
ISP1:
10Gbps residential (unshared), unmetered and unlimited, 1 static IPv4, a /56
block of IPv6.
Pros: high speed and unmetered. Serving 5TB/day on average now, peak traffic
55TB/day.
Cons: the ISP doesn't support rDNS.
ISP2:
1Gbps semi-business-residential (fiber shared with the apartment building),
data caps at 1TB per month, a /28 block of IPv4 and a /56 block of IPv6.
Pros: the ISP supports rDNS with IPv4. Suitable for name servers.
Cons: speed severely throttles after 1TB/month.
Hostname: h12ssl-nt
CPU: AMD EPYC 75F3, 32 cores
RAM: 8x64=512GB DDR4 ECC RDIMM
Motherboard: Supermicro H12SSL-NT
Storage:
boot drives: 2x 1.6TB Kioxia CM6-V, PCIe Gen 4 NVMe SSD, ZFS RAID1, with an
Intel Optane P1600X 58GB, PCIe Gen3 NVMe SSD as SLOG
general purpose storage: 10x 7.68TB Kioxia CD6-R, PCIe Gen 4 NVNe SSD, ZFS
RAIDz2, with an Intel Optane P1600X 58GB, PCIe Gen3 NVMe SSD as SLOG
Description:
This server runs an installation of Proxmox VE in an ATX PC case. Most of the
components were bought used from the second hand market, except for the two
Intel Optane SSDs. It provides many services important to the community and
essential to my digital life, such as:
- 1 node of repo.jing.rocks (the largest free software mirror in Japan, no
kidding)
- 1 authoritative name server and 2 recursive DNS servers (pihole, with DHCP)
- 1 MariaDB cluster node, 2 Postgresql cluster nodes for various services
- 1 node of invidious.jing.rocks (luckily has not been banned by google)
- mastodon.jing.lgbt (1 node for load-balancing)
- multiple web servers in LXC containers
- 1 Proxmox Mail Gateway cluster node
It's also worth mentioning that I provide {web,name,mail} servers for Dragora
GNU/Linux-libre.
Pros:
It's very stable, high performance, is also used as a NAS and a build machine.
Cons:
Has the AMD equivalent of Intel ME (forgot the name...). Has onboard non-free
IPMI but not connected to the internet. Built this machine when I didn't really
know free software. Also it's at risk of running out of RAM because of ZFS
filesystem caching (which Bob disagrees)
Hostname: rome2d16-2t
CPU: 2 sockets: AMD EPYC 7773X, 2x64=128 cores
RAM: 16x64=1024GB DDR4 ECC RDIMM
Motherboard: Asrock ROME2D16-2T
Storage:
boot drives: 2x 1.6TB Kioxia CM6-V, PCIe Gen 4 NVMe SSD, ZFS RAID1, with an
Intel Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
general purpose storage:
- 11x 16TB SATA spinning hard drives, ZFS RAIDz3, with an L2ARC of 3.84TB
SATA SSD, with a three-way mirrored special metadata vdev 1.92TB, with an Intel
Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
- 11x 18TB SATA spinning hard drives, ZFS RAIDz3, with an L2ARC of 3.84TB
SATA SSD, with a three-way mirrored special metadata vdev 1.92TB, with an Intel
Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
- 2x 8TB SATA spinning hard drives, ZFS RAID1, with an Intel Optane P1600X
118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
- 2x 3.84TB SATA SSD, ZFS RAID1, with an Intel Optane P1600X 118GB, PCIe Gen3
NVMe SSD as SLOG (shared), reserved for gcc compile farm
Description:
This server runs an installation of Proxmox VE in an EATX PC case. Many
components were bought used. It mainly provides these services:
- 3 VMs for gcc compile farm [1]: {cfarm420,cfarm421,cfarm422}.cfarm.net (I
specifically asked for those host names :)
- 1 node of repo.jing.rocks
- 1 authoritative name server and 1 internal DNS server (pihole, with DHCP)
- 1 vcs server that runs a forgejo instance and a savane instance (testing
only)
- 1 MariaDB cluster node, 2 Postgresql cluster nodes for various services
- mastodon.jing.lgbt (1 node for load-balancing)
- 1 Proxmox Mail Gateway cluster node
Pros:
Suitable for highly parallel workload. The second ZFS pool has about 50TB of
space available, while others are at about 75~80% capacity.
Cons:
Has the AMD equivalent of Intel ME (forgot the name...). Has onboard non-free
IPMI but not connected to the internet. Built this machine when I didn't really
know free software. Also it's at risk of running out of RAM because of ZFS
filesystem caching (which Bob disagrees).
[1] https://portal.cfarm.net/machines/list/
Hostname: z490
CPU: Intel Core i9-10900K
RAM: 4x32=128GB DDR4 UDIMM
Description:
It also runs Proxmox VE, but mainly for more "not highly parallel" services,
like jitsi, nextcloud, invidious. About to be turned into a second-level L3
switch. It has Intel ME.
Hostname: x570d4u-2l2t
CPU: AMD Ryzen 9 5950X, 32 cores.
RAM: 4x32=128GB DDR4 ECC UDIMM
Motherboard: Asrock X570D4U-2L2T
Description:
It also runs Proxmox VE, but mainly functions as a router/core switch. It's
super stable, has a long uptime. It has 10 10GbE ports in total. It runs 2
OpenWRT VMs, 1 ZNC IRC bouncer, 1 Proxmox Mail Gateway cluster node, and 1
reverse proxy/load-balancer for all web servers. Has a 4x960GB SATA SSD ZFS
RAID0 that isn't storing anything...
Pros: It boots really fast.
Cons:
Has the AMD equivalent of Intel ME (forgot the name...). Has onboard non-free
IPMI but not connected to the internet. The boot drives are about to be
replaced with ZFS RAID1, so a reinstall is coming.
Hostname: altrad8ud-1l2t
CPU: Ampere Altra Max Q128-30 engineering sample, 128 cores.
RAM: 8x16=128GB for now, upgrading to 512GB
Motherboard: Asrock ALTRAD8UD-1L2T
Description:
A new machine that is not x86! It runs a version of Proxmox VE that I patched
and built from source then ported to arm64. It runs Trisquel perfectly in my
testing. Currently not running any service. About to be reinstalled, replacing
btrfs raid1 with zfs raid1.
Pros: 100% free. Suitable for highly parallel workload, preparing to offer VMs
to Trisquel build farm and gcc compile farm. Runs really cool.
Cons: Asrock ships a non-free UEFI/BIOS and a non-free distribution of OpenBMC
(not connected to internet).
Hostname: (none)
CPU: Rockchip RK3588
RAM: 32GB LPDDR4
Storage: 4TB NVMe SSD, consumer grade, and onboard eMMC and/or microSD cards.
Description:
I also have 3 rockchip rk3588 based SBCs. They have my patched Proxmox VE
arm64 installed, with a custom kernel build. They all can run linux-libre
without framebuffer support (I gave up on those a long time ago). They are
under powered for compiling jobs, but suitable for server use.
Pros: Stable and power efficient.
Cons:
Has a fatal flaw that requires a non-free blob to boot. The DDR init blob must
be inserted into u-boot, or else it doesn't boot. Currently using a custom
u-boot build. I hope maybe someone can reverse engineer it someday...
Some useful URLs:
https://stats.jing.rocks/
https://munin.jing.rocks/+(add "munin" or "munin/", want to avoid scraping bots
here)
https://git.jing.rocks/cgit/home-config.git/
https://goaccess.jing.rocks/
Thanks for reading, looking forward to your opinions. I'm going back to patch
proxmox and linux...
--
Jing Luo
About me: https://jing.rocks/about/
GPG Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC
signature.asc
Description: OpenPGP digital signature
