So I wanted to know if Andrew's changes in 2.3 are going to have a noticeable affect at scale on Leadership. So I went and set up a test with HA controllers running 10 machines each with 3 containers, and then distributing ~500 applications each with 3 units across everything. I started at commit 2e50e5cf4c3 which is just before Andrew's Lease patch landed.
juju bootstrap aws/eu-west-2 --bootstrap-constraint instance-type=m4.xlarge --config vpc-id=XXXX juju enable-ha -n3 # Wait for things to stablize juju deploy -B cs:~jameinel/ubuntu-lite -n10 --constraints instance-type=m4.xlarge # wait #set up the containers for i in `seq 0 9`; do juju deploy -n3 -B cs:~jameinel/ubuntu-leader ul --to lxd:${i},lxd:${i},lxd:${i} done # scale up. I did this more in batches of a few at a time, but slowly grew all the way up for j in `seq 1 49`; do echo $j for i in `seq 0 9`; do juju deploy -B -n3 cs:~jameinel/ubuntu-leader ul${i}{$j} --to ${i}/lxd/0,${i}/lxd/1,${i}/lxd/2 & done time wait done I let it go for a while until "juju status" was happy that everything was up and running. Note that this was 1500 units, 500 applications in a single model. time juju status was around 4-10s. I was running 'mongotop' and watching 'top' while it was running. I then upgraded to the latest juju dev (c49dd0d88a). Now, the controller immediately started thrashing, with bad lease documents in the database, and eventually got to the point that it ran out of open file descriptors. Theoretically upgrading 2.2 => 2.3 won't have the same problem because the actual upgrade step should run. However, if I just did "db.leases.remove({})" it recovered. I ended up having to restart mongo and jujud to recover from the open file handles, but it did eventually recover. At this point, I waited again for everything to look happy, and watch mongotop and top again. These aren't super careful results, where I would want to run things for like an hour each and check the load over that whole time. Really I should have set up prometheus monitoring. But as a quick check, these are the top values for mongotop before: ns total read write local.oplog.rs 181ms 181ms 0ms juju.txns 120ms 10ms 110ms juju.leases 80ms 34ms 46ms juju.txns.log 24ms 4ms 19ms ns total read write local.oplog.rs 208ms 208ms 0ms juju.txns 140ms 12ms 128ms juju.leases 98ms 42ms 56ms juju.charms 43ms 43ms 0ms ns total read write local.oplog.rs 220ms 220ms 0ms juju.txns 161ms 14ms 146ms juju.leases 115ms 52ms 63ms presence.presence.beings 69ms 68ms 0ms ns total read write local.oplog.rs 213ms 213ms 0ms juju.txns 164ms 15ms 149ms juju.leases 82ms 35ms 47ms presence.presence.beings 79ms 78ms 0ms ns total read write local.oplog.rs 221ms 221ms 0ms juju.txns 168ms 13ms 154ms juju.leases 95ms 40ms 55ms juju.statuses 33ms 16ms 17ms totals: 1043 local.oplog.rs juju.txns 868 juju.leases 470 and after ns total read write local.oplog.rs 95ms 95ms 0ms juju.txns 68ms 6ms 61ms juju.leases 33ms 13ms 19ms juju.txns.log 13ms 3ms 10ms ns total read write local.oplog.rs 200ms 200ms 0ms juju.txns 160ms 10ms 150ms juju.leases 78ms 35ms 42ms juju.txns.log 29ms 4ms 24ms ns total read write local.oplog.rs 151ms 151ms 0ms juju.txns 103ms 6ms 97ms juju.leases 45ms 20ms 25ms juju.txns.log 21ms 6ms 15ms ns total read write local.oplog.rs 138ms 138ms 0ms juju.txns 98ms 6ms 91ms juju.leases 30ms 13ms 16ms juju.txns.log 18ms 3ms 14ms ns total read write local.oplog.rs 218ms 218ms 0ms juju.txns 196ms 14ms 182ms juju.leases 81ms 36ms 44ms juju.txns.log 34ms 5ms 29ms
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev