Re: Notes from Scale testing

2013-10-31 Thread William Reade
On Wed, Oct 30, 2013 at 9:26 AM, Gustavo Niemeyer gust...@niemeyer.netwrote: On Wed, Oct 30, 2013 at 7:57 AM, John Arbash Meinel j...@arbash-meinel.com wrote: Where is N^2 coming from? If you add N units one-by-one each new add triggers all existing units to wake up and ask for the Life

Re: Notes from Scale testing

2013-10-31 Thread William Reade
On Thu, Oct 31, 2013 at 12:37 AM, William Reade william.re...@canonical.com wrote: The service watch fires on any change to the document, and adding a unit increments the service's unitcount field. The easy fix is to wrap the service watcher so that it discards irrelevant changes; the hard

Notes from Scale testing

2013-10-30 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm trying to put together a quick summary of what I've found out so far with testing juju in an environment with thousands (5000+) agents. 1) I didn't ever run into problems with connection failures due to socket exhaustion. The default upstart

Re: Notes from Scale testing

2013-10-30 Thread Gustavo Niemeyer
On Wed, Oct 30, 2013 at 6:23 AM, John Arbash Meinel j...@arbash-meinel.com wrote: I'm trying to put together a quick summary of what I've found out so far with testing juju in an environment with thousands (5000+) agents. Great testing, John. 2) Agents seem to consume about 17MB resident

Re: Notes from Scale testing

2013-10-30 Thread Nate Finch
On Wed, Oct 30, 2013 at 9:23 AM, John Arbash Meinel j...@arbash-meinel.comwrote: 2) Agents seem to consume about 17MB resident according to 'top'. That should mean we can run ~450 agents on an m1.large. Though in my testing I was running ~450 and still had free memory, so I'm guessing there

Re: Notes from Scale testing

2013-10-30 Thread Kapil Thangavelu
Hi John, This is awesome, its great to see this scale testing and analysis. Some additional questions/comments inline. On Wed, Oct 30, 2013 at 6:23 AM, John Arbash Meinel j...@arbash-meinel.comwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm trying to put together a quick summary

Re: Notes from Scale testing

2013-10-30 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... - From what I can tell, all units take out a watch on their service so that they can monitor its Life and CharmURL. However, adding a unit to a service triggers a change on that service, even though Life and CharmURL haven't changed. If we

Re: Notes from Scale testing

2013-10-30 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... 4) If I bring up the units one by one (for i in `seq 500`; do for j in `seq 10` do juju add-unit --to $j ; time wait; done), it ends up triggering O(N^2) behavior in the system. Each unit agent seems to have a watcher for other units of

Re: Notes from Scale testing

2013-10-30 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-10-30 18:11, Nate Finch wrote: On Wed, Oct 30, 2013 at 9:23 AM, John Arbash Meinel j...@arbash-meinel.com mailto:j...@arbash-meinel.com wrote: 2) Agents seem to consume about 17MB resident according to 'top'. That should mean we can

Re: Notes from Scale testing

2013-10-30 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... The log size didn't come up again in this email. Not sure if you meant separately or just got lost in the message length. I didn't explicitly enumerate it, but it is because of section (4). Namely, bringing up 1000 units triggers 1000*1000