On Wed, Oct 30, 2013 at 6:23 AM, John Arbash Meinel <j...@arbash-meinel.com> wrote: > I'm trying to put together a quick summary of what I've found out so > far with testing juju in an environment with thousands (5000+) agents.
Great testing, John. > 2) Agents seem to consume about 17MB resident according to 'top'. That > should mean we can run ~450 agents on an m1.large. Though in my > testing I was running ~450 and still had free memory, so I'm guessing > there might be some copy-on-write pages (17MB is very close to the > size of the jujud binary). Yeah, RSS is not straightforward to measure correctly. The crude readings are pretty much always overestimated. > 4) If I bring up the units one by one (for i in `seq 500`; do for j in > `seq 10` do juju add-unit --to $j &; time wait; done), it ends up > triggering O(N^2) behavior in the system. Each unit agent seems to > have a watcher for other units of the same service. So when you add 1 > unit, it wakes up all existing units to let them know about it. In > theory this is on a 5s rate limit (only 1 wakeup per 5 seconds). In > practice it was taking >3s per add unit call [even when requesting > them in parallel]. I think this was because of the load on the API > server of all the other units waking up and asking for details at the > same time. In theory answering all watching questions should eventually be very cheap, if it isn't right now. The yes/no data for several thousand units easily fits all in memory, and the API servers also learn about changes as they go through to the database, so there's no big reason to touch the database for these operations. Caching will also play a larger role inside the API servers as juju moves towards scalability. > - From what I can tell, all units take out a watch on their service so > that they can monitor its Life and CharmURL. However, adding a unit to > a service triggers a change on that service, even though Life and > CharmURL haven't changed. If we split out Watching the > units-on-a-service from the lifetime and URL of a service, we could > avoid the thundering N^2 herd problem while starting up a bunch of > units. Though UpgradeCharm is still going to thundering herd. Where is N^2 coming from? > It then seems to restart the Unit agent, which goes through the steps > of making all the same requests again. (Get the Life of my Unit, get > the Life of my service, get the UUID of this environment, etc., there > are 41 requests before it gets to APIAddress) Ugh! > I would be fine doing max(1, NumCPUs()-1) or something similar. I'd > rather do it inside jujud rather than in the cloud-init script, > because computing NumCPUs is easier there. But we should have *a* way > to scale up the central node that isn't just scaling out to more API > servers. NumCPUs sounds like a fine initial setting. > I certainly think we need a way to scale Mongo as well. If it is just > 1 CPU per connection then scaling horizontally with API servers should > get us around that limit. It's one thread per connection. > 10) Allowing "juju add-unit -n 100 --to X" did make things a lot > easier to bring up. Though it still takes a while for the request to > finish. It felt like the api call triggered work to start happening in > the background which made the current api call take longer to finally > complete. (as in, minutes once we had >1000 units). It doesn't sound like optimizing for a huge volume of immediate units is an important goal, other than for streamlining that kind of testing. I have very rarely observed use cases that do that, or that have resources available at all for doing that. Even large providers will generally deny requests for large deltas at once, due to their controlled growth schedules. It sounds more relevant to have juju able to cope with these loads effectively and timely once they do reach such scales. > Not everything in there is worth landing in trunk (rudimentary API > caching, etc). > > That's all I can think of for now, though I think there is more to be > explored. Again, well done. gustavo @ http://niemeyer.net -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
