Thank you mikael and jake, matt and matthew and richard! (and jon, and dev for trying)
the linode bill is now paid. For the record, peak "earnings" for the patreon contribution page was $212 in may 2016, and fell below $100 in May, 2018. I forget when I scaled the number of servers back in between those times. In cerowrt's glory days we were burning ~2k a month on the openwrt build farm, which I had to quit doing 6 months after the google grant did. Prior to that we had donated servers from ISC, which had to get out of the free ISP business. I kind of miss running on bare metal, with tons of disk space. Cloud services are pretty cheap, cloud storage isn't. Oy! I remember the headaches and hassles when the number of openwrt buildslaves dropped to 1 for 6 weeks during a critical phase.... Thankfully that farm looks healthy at the moment: http://phase1.builds.openwrt.org/buildslaves Costs on the "flent-farm" continue to drop. Our earliest linode servers cost $20/month and our two latest ones (nanoservers) cost $5/month. For "science!" I've been generally unwilling to update/change these much, the most critical keeping the same kernel versions they had for the last couple years. I note that linode at least a year+ ago, started defaulting to a kernel with fq_codel enabled *by default*, bql just works, irtt, flent, netperf all "just install" from apt, etc, etc. It's currently a matter of a few minutes to get a basic flent server running in the cloud, and sometimes I wish we had more worldwide coverage - australia - aws - france - china - It turns out arm based servers are quite cheap nowadays... yep, here I am 6 bucks to the good and trying to figure out another server on which to spend it. :) I have not been maintaining the flent network all that well, but your support inspired me to go fix two of them. flent-singapore.taht.net: the newest of these servers de.taht.net: the second newest now support BBR, cubic, and reno, and have irtt running with support for 1ms intervals. (thx pete for showing me how on flent-london.bufferbloat.net) Linux flent-singapore.taht.net 4.18.16-x86_64-linode118 #1 SMP PREEMPT Mon Oct 29 15:38:25 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Linux de.taht.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux In the teklibre sub-basement are a few dozen tiny openwrt/armbian boxes, and two honkin, but ancient 12 core xeon servers ("spaceheater" and "ceres") with 10TB of storage. I keep a variety of vms on those in complicated network emulation setups, an indexed copy of all the source code in the world, my openwrt build system, etc. and here's a puzzler for you! Both boxes are running ntp yet one box was still *30* seconds off. irtt 1ms between california and germany has a mindboggling amount of loss... in america... ahh... tools... 5. be-3651-cr02.sunnyvale.ca.ibone.comcast.net 71.9% 33 11.4 11.6 9.4 16.0 2.3 6. be-11083-pe02.529bryant.ca.ibone.comcast.net 56.2% 33 13.1 11.3 9.5 14.8 1.6 Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 166.2ms 175.4ms 174.3ms 232.7ms 4.85ms send delay 27.9s 27.9s 27.9s 28s 4.07ms receive delay -27.7s -27.7s -27.7s -27.7s 1.98ms IPDV (jitter) 434ns 1.46ms 979µs 37.64ms 1.78ms send IPDV 408ns 1.36ms 954µs 34.92ms 1.51ms receive IPDV 2ns 378µs 72.2µs 32.98ms 1.08ms send call time 3.55µs 14.9µs 1.61ms 16.7µs timer error 0s 19.6µs 5.52ms 68.3µs server proc. time 710ns 7.67µs 10.33ms 91.6µs duration: 1m1s (wait 698.1ms) packets sent/received: 58852/44855 (23.78% loss) server packets received: 45579/58852 (22.55%/1.59% loss up/down) bytes sent/received: 3531120/2691300 send/receive rate: 470.8 Kbps / 358.8 Kbps packet length: 60 bytes timer stats: 1148/60000 (1.91%) missed, 1.96% error restarted ntp, and: Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 166.3ms 175.9ms 175.1ms 233ms 4.78ms send delay 83.97ms 92.57ms 91.9ms 137.4ms 4.1ms receive delay 80.86ms 83.32ms 82.86ms 137.4ms 1.89ms IPDV (jitter) 316ns 1.44ms 977µs 63.68ms 1.75ms send IPDV 41ns 1.35ms 933µs 35.49ms 1.5ms receive IPDV 2ns 345µs 83.8µs 55.06ms 971µs send call time 3.55µs 16.4µs 1.07ms 11.8µs timer error 1ns 23.4µs 5.02ms 76.8µs server proc. time 716ns 8.22µs 13.13ms 117µs duration: 1m1s (wait 699.1ms) packets sent/received: 58779/45154 (23.18% loss) server packets received: 45912/58779 (21.89%/1.65% loss up/down) bytes sent/received: 3526740/2709240 send/receive rate: 470.2 Kbps / 361.2 Kbps packet length: 60 bytes timer stats: 1221/60000 (2.04%) missed, 2.34% error Verses singapore. Min Mean Median Max Stddev --- ---- ------ --- ------ RTT 173.6ms 179.2ms 178.7ms 194.7ms 2.5ms send delay 113ms 118.5ms 118ms 131.9ms 2.32ms receive delay 59.7ms 60.68ms 60.45ms 74.75ms 980µs IPDV (jitter) 203ns 1.24ms 978µs 18.8ms 1.01ms send IPDV 20ns 1.19ms 983µs 14ms 881µs receive IPDV 0s 195µs 20.3µs 14.11ms 539µs send call time 3.67µs 14.7µs 1.75ms 13.2µs timer error 1ns 21.4µs 7.82ms 93.9µs server proc. time 680ns 3.27µs 435µs 9.35µs duration: 1m1s (wait 584ms) packets sent/received: 58709/55848 (4.87% loss) server packets received: 55848/58709 (4.87%/0.00% loss up/down) bytes sent/received: 3522540/3350880 send/receive rate: 469.7 Kbps / 446.8 Kbps packet length: 60 bytes timer stats: 1290/59999 (2.15%) missed, 2.14% error -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740 _______________________________________________ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat