Re: testing ejabberd
Gary C Martin wrote: http://wiki.laptop.org/go/Ejabberd_resource_tests#Try_4:_a_few_thousand_users One extra figure that would be interesting is the server response latency to client requests, not sure if hyperactivity gives you that easily. No, I don't think hyperactivity does measure latency. The only metric I have is that activity sharing between XOs worked well enough while competing with 2000 hyperactivity clients. There are other test suites that claim to measure latency, so I'm looking into running one of them at the same time. Douglas ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Server-devel] testing ejabberd
Gary C Martin wrote: http://wiki.laptop.org/go/Ejabberd_resource_tests#Try_4:_a_few_thousand_users One extra figure that would be interesting is the server response latency to client requests, not sure if hyperactivity gives you that easily. No, I don't think hyperactivity does measure latency. The only metric I have is that activity sharing between XOs worked well enough while competing with 2000 hyperactivity clients. There are other test suites that claim to measure latency, so I'm looking into running one of them at the same time. Douglas ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: testing ejabberd
Hi Douglas, On 4 Oct 2008, at 05:30, Douglas Bagnall wrote: The results are summarised here: http://wiki.laptop.org/go/ Ejabberd_resource_tests#Try_4:_a_few_thousand_users In short, with 1GB ram, ejabberd coped with a stable load of 2000 connections, but it went crazy when faced with more, bouncing off the RAM ceiling, dropping clients, and freezing its web admin interface. Then after a quiet period it recuperated and gamely made the fatal number of connections. Hey, nice set of graphs. So, my take on looking through them was that you are, most definitely, memory bound. Once you get to ~1500 active users you're out of physical memory, IO waits go up, CPU load goes down, VM usage spikes, things that really shouldn't start getting swapped to disk, services start grinding to a halt left, right, and centre... I had a colleague who's answer (in part) to pretty well any server load issue that came up was add RAM. He was right, most of the time, and adding ram never hurt anyone :-) One extra figure that would be interesting is the server response latency to client requests, not sure if hyperactivity gives you that easily. Adding RAM should help to stabilise the system behaviour, but you'll start drifting into server responses that are slow enough that various layers of the communication stack start to think things have broken. At that point there will likely be a spike in retransmissions/ reconnections et al, and all that extra traffic will quickly bring the server to it's knees again. :-) All interesting stuff. --Gary ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: testing ejabberd
I wrote: I've written up my recent testing of ejabberd for the wiki: http://wiki.laptop.org/go/Ejabberd_resource_tests It is not completely satisfactory: I don't have the resources to test up to 3000 active users which I believe is an important target. At lower numbers, however, ejabberd's memory consumption seems to be linear, and it looks to be roughly the case that 0.5 GB per 1000 users is enough. (Just barely -- that's a limit, not a recommendation). Since then, thanks to hyperactivity pointers from Guillaume, I got ejabberd to very briefly accept about 4700 connections, and almost simultaneously, to crash. I'm quite pleased with this on both counts, even though, because it happened during the period late on Fridays that our host company offers free beer upstairs, I did not actually witness the events. The results are summarised here: http://wiki.laptop.org/go/Ejabberd_resource_tests#Try_4:_a_few_thousand_users In short, with 1GB ram, ejabberd coped with a stable load of 2000 connections, but it went crazy when faced with more, bouncing off the RAM ceiling, dropping clients, and freezing its web admin interface. Then after a quiet period it recuperated and gamely made the fatal number of connections. From time to time ejabberd logged errors or warnings but they don't seem to relate to much. I'm trying to get this automated enough so I can leave it running in the background and think of something else. Douglas ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Server-devel] testing ejabberd
I wrote: I've written up my recent testing of ejabberd for the wiki: http://wiki.laptop.org/go/Ejabberd_resource_tests It is not completely satisfactory: I don't have the resources to test up to 3000 active users which I believe is an important target. At lower numbers, however, ejabberd's memory consumption seems to be linear, and it looks to be roughly the case that 0.5 GB per 1000 users is enough. (Just barely -- that's a limit, not a recommendation). Since then, thanks to hyperactivity pointers from Guillaume, I got ejabberd to very briefly accept about 4700 connections, and almost simultaneously, to crash. I'm quite pleased with this on both counts, even though, because it happened during the period late on Fridays that our host company offers free beer upstairs, I did not actually witness the events. The results are summarised here: http://wiki.laptop.org/go/Ejabberd_resource_tests#Try_4:_a_few_thousand_users In short, with 1GB ram, ejabberd coped with a stable load of 2000 connections, but it went crazy when faced with more, bouncing off the RAM ceiling, dropping clients, and freezing its web admin interface. Then after a quiet period it recuperated and gamely made the fatal number of connections. From time to time ejabberd logged errors or warnings but they don't seem to relate to much. I'm trying to get this automated enough so I can leave it running in the background and think of something else. Douglas ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: testing ejabberd
I've written up my recent testing of ejabberd for the wiki: http://wiki.laptop.org/go/Ejabberd_resource_tests It is not completely satisfactory: I don't have the resources to test up to 3000 active users which I believe is an important target. At lower numbers, however, ejabberd's memory consumption seems to be linear, and it looks to be roughly the case that 0.5 GB per 1000 users is enough. (Just barely -- that's a limit, not a recommendation). With 1200 users making some communication every 15 seconds, the 2GHz dual core pentium was bouncing along with a load average around 2 and ejabberd over 100% CPU usage. I don't know whether 15 seconds is a reasonable interval: if e.g. each keystroke in a shared Write touches ejabberd, then 15 seconds seems long; otherwise perhaps it's very short. Once I realised that the open files resource limit was killing ejabberd (which took an embarrassingly long time, not helped by cryptic log messages), it was stable under all loads. From time to time I tried sharing activities between XOs and they were always responsive. Douglas ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Server-devel] testing ejabberd
I've written up my recent testing of ejabberd for the wiki: http://wiki.laptop.org/go/Ejabberd_resource_tests It is not completely satisfactory: I don't have the resources to test up to 3000 active users which I believe is an important target. At lower numbers, however, ejabberd's memory consumption seems to be linear, and it looks to be roughly the case that 0.5 GB per 1000 users is enough. (Just barely -- that's a limit, not a recommendation). With 1200 users making some communication every 15 seconds, the 2GHz dual core pentium was bouncing along with a load average around 2 and ejabberd over 100% CPU usage. I don't know whether 15 seconds is a reasonable interval: if e.g. each keystroke in a shared Write touches ejabberd, then 15 seconds seems long; otherwise perhaps it's very short. Once I realised that the open files resource limit was killing ejabberd (which took an embarrassingly long time, not helped by cryptic log messages), it was stable under all loads. From time to time I tried sharing activities between XOs and they were always responsive. Douglas ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: testing ejabberd
Le mercredi 24 septembre 2008 à 16:29 +1200, Douglas Bagnall a écrit :
Guillaume,
Would be helpful if you could upload Gabble log somewhere. Before
starting hyperactivity, launch Gabble manually like this:
GABBLE_PERSIST=1 GABBLE_LOGFILE=/tmp/gabble.log GABBLE_DEBUG=all
LM_DEBUG=net /usr/lib/telepathy/telepathy-gabble
Thanks. That was enough for me to sort it out -- the problem was
caused by ejabberd restricting the number of registrations per IP
address. Adding {registration_timeout, infinity}. to ejabberd.cfg
fixed it.
I've put the log at http://halo.gen.nz/gabble-wired-connection-1.log
but only in case you are curious.
I've tested up to about 350 users from various machines at various
activity rates. Collaboration continues to work while ejabberd is
under this load, while its memory use grows to around 160MB. I'll
report on this in more detail soon.
Oh right, I already observed this problem too. I added [1] explanations
in hyperactivity's README so hopefully you should be the last person
confused by this problem. :)
G.
[1]
https://dev.laptop.org/git?p=users/guillaume/hyperactivity/.git;a=commitdiff;h=e50d399881b823cad51edcc5525d53a972968459
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
testing ejabberd
hi
I'm having some trouble using hyperactivity to test ejabberd.
Hyperactivity always ends up looping over unsuccessful accounts,
producing output like this:
can't connect hyperactivity-ac4ec2e2-892e-11dd-a4b7-0017c40d34e4. Remove it
have to create 1 accounts
create
accounts/gabble/schoolserver.dell.xs.laptop.org/hyperactivity-ac9cecec-892e-11dd-a4b7-0017c40d34e4.account
can't connect hyperactivity-ac6d40aa-892e-11dd-a4b7-0017c40d34e4. Remove it
have to create 1 accounts
create
accounts/gabble/schoolserver.dell.xs.laptop.org/hyperactivity-acb4ce02-892e-11dd-a4b7-0017c40d34e4.account
can't connect hyperactivity-ac85168a-892e-11dd-a4b7-0017c40d34e4. Remove it
What ejabberd says of each of these is something like:
I(0.258.0:ejabberd_listener:112) : (#Port0.464) Accepted
connection {{0,0,0,0,0,65535,44050,2588},33012} -
{{0,0,0,0,0,65535,44050,1},5222}
This would make simple sense if hyperactivity didn't succeed every now
or then. These usable accounts build up over time, so hyperactivity
ends up starting with a few of them. So in the sea of unsuccessful
creations there is every now and then a line like:
client hyperactivity-c3e52044-88f3-11dd-a913-0017c40d34e4: -- change
current activity
Although that has no server-side correspondent. The anomalous messages
on the server side are:
=INFO REPORT 2008-09-23 01:15:34 ===
I(0.386.0:ejabberd_c2s:478) :
({socket_state,gen_tcp,#Port0.451,0.385.0}) Failed legacy
authentication for
[EMAIL PROTECTED]/Telepathy
=INFO REPORT 2008-09-23 01:15:34 ===
I(0.388.0:ejabberd_c2s:438) :
({socket_state,gen_tcp,#Port0.453,0.387.0}) Accepted legacy
authentication for
[EMAIL PROTECTED]/Telepathy
=INFO REPORT 2008-09-23 01:15:34 ===
I(0.388.0:mod_shared_roster:640) : user_available for
hyperactivity-c3e52044-88f3-11dd-a913-0017c40d34e4 @
schoolserver.dell.xs.laptop.org (1 resources)
[ ... millions of the 'Accepted connection' messages, then ... ]
=INFO REPORT 2008-09-23 01:15:54 ===
I(0.388.0:ejabberd_c2s:1290) :
({socket_state,gen_tcp,#Port0.453,0.387.0}) Close session for
[EMAIL PROTECTED]/Telepathy
=INFO REPORT 2008-09-23 01:15:54 ===
I(0.388.0:mod_shared_roster:679) : unset_presence for
hyperactivity-c3e52044-88f3-11dd-a913-0017c40d34e4 @
schoolserver.dell.xs.laptop.org / Telepathy - [] (0 resources)
Has somebody seen this before? What am I doing wrong?
Douglas
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Re: testing ejabberd
Le mardi 23 septembre 2008 à 18:22 +1200, Douglas Bagnall a écrit :
hi
I'm having some trouble using hyperactivity to test ejabberd.
Hyperactivity always ends up looping over unsuccessful accounts,
producing output like this:
can't connect hyperactivity-ac4ec2e2-892e-11dd-a4b7-0017c40d34e4. Remove it
have to create 1 accounts
create
accounts/gabble/schoolserver.dell.xs.laptop.org/hyperactivity-ac9cecec-892e-11dd-a4b7-0017c40d34e4.account
can't connect hyperactivity-ac6d40aa-892e-11dd-a4b7-0017c40d34e4. Remove it
have to create 1 accounts
create
accounts/gabble/schoolserver.dell.xs.laptop.org/hyperactivity-acb4ce02-892e-11dd-a4b7-0017c40d34e4.account
can't connect hyperactivity-ac85168a-892e-11dd-a4b7-0017c40d34e4. Remove it
What ejabberd says of each of these is something like:
I(0.258.0:ejabberd_listener:112) : (#Port0.464) Accepted
connection {{0,0,0,0,0,65535,44050,2588},33012} -
{{0,0,0,0,0,65535,44050,1},5222}
This would make simple sense if hyperactivity didn't succeed every now
or then. These usable accounts build up over time, so hyperactivity
ends up starting with a few of them. So in the sea of unsuccessful
creations there is every now and then a line like:
client hyperactivity-c3e52044-88f3-11dd-a913-0017c40d34e4: -- change
current activity
Although that has no server-side correspondent. The anomalous messages
on the server side are:
=INFO REPORT 2008-09-23 01:15:34 ===
I(0.386.0:ejabberd_c2s:478) :
({socket_state,gen_tcp,#Port0.451,0.385.0}) Failed legacy
authentication for
[EMAIL PROTECTED]/Telepathy
=INFO REPORT 2008-09-23 01:15:34 ===
I(0.388.0:ejabberd_c2s:438) :
({socket_state,gen_tcp,#Port0.453,0.387.0}) Accepted legacy
authentication for
[EMAIL PROTECTED]/Telepathy
=INFO REPORT 2008-09-23 01:15:34 ===
I(0.388.0:mod_shared_roster:640) : user_available for
hyperactivity-c3e52044-88f3-11dd-a913-0017c40d34e4 @
schoolserver.dell.xs.laptop.org (1 resources)
[ ... millions of the 'Accepted connection' messages, then ... ]
=INFO REPORT 2008-09-23 01:15:54 ===
I(0.388.0:ejabberd_c2s:1290) :
({socket_state,gen_tcp,#Port0.453,0.387.0}) Close session for
[EMAIL PROTECTED]/Telepathy
=INFO REPORT 2008-09-23 01:15:54 ===
I(0.388.0:mod_shared_roster:679) : unset_presence for
hyperactivity-c3e52044-88f3-11dd-a913-0017c40d34e4 @
schoolserver.dell.xs.laptop.org / Telepathy - [] (0 resources)
Has somebody seen this before? What am I doing wrong?
Would be helpful if you could upload Gabble log somewhere. Before
starting hyperactivity, launch Gabble manually like this:
GABBLE_PERSIST=1 GABBLE_LOGFILE=/tmp/gabble.log GABBLE_DEBUG=all
LM_DEBUG=net /usr/lib/telepathy/telepathy-gabble
(Gabble path can change depending on your distro).
You should have logs in /tmp/gabble.log
G.
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Re: testing ejabberd
Guillaume,
Would be helpful if you could upload Gabble log somewhere. Before
starting hyperactivity, launch Gabble manually like this:
GABBLE_PERSIST=1 GABBLE_LOGFILE=/tmp/gabble.log GABBLE_DEBUG=all
LM_DEBUG=net /usr/lib/telepathy/telepathy-gabble
Thanks. That was enough for me to sort it out -- the problem was
caused by ejabberd restricting the number of registrations per IP
address. Adding {registration_timeout, infinity}. to ejabberd.cfg
fixed it.
I've put the log at http://halo.gen.nz/gabble-wired-connection-1.log
but only in case you are curious.
I've tested up to about 350 users from various machines at various
activity rates. Collaboration continues to work while ejabberd is
under this load, while its memory use grows to around 160MB. I'll
report on this in more detail soon.
Douglas
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
