Re: [jabberd2] Clustering
Dnia 24-10-2007, Śr o godzinie 07:41 +0200, emmanuel giraud pisze: > > I have a question about a way to implement of clustering in > jabberd2. > > Why don't we use software open source that run clusters? > > It may be stupid but what's your opinion? > > Why do you developp another solutions made by others? > > http://openmosix.sourceforge.net/ Few reasons: - jabberd 2 daemons are single-threaded and will not distribute over mosix and alike clusters - in case of one cluster component crash we want others to take over, not crash also on every cluster node - jabberd 2 processes may grow large and would generate very heavy synchronization load, that would effectively jeopardize the gain - we do not want process-level clustering overhead (see previous point) -- /\_./o__ Tomasz Sterna (/^/(_^^' Xiaoka.com ._.(_.)_ XMPP: [EMAIL PROTECTED] ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 emmanuel giraud schrieb: >> Hi, >> >> I have a question about a way to implement of clustering in jabberd2. >> Why don't we use software open source that run clusters? >> It may be stupid but what's your opinion? >> >> Why do you developp another solutions made by others? >> >> http://openmosix.sourceforge.net/ Because you are comparing apples (no, not the computers as google might suggest) and oranges ;-) SCNR - -- \\\||/// \\ - - // ( @ @ ) - -oOo--( )--oOo--- Bernd Holzmüller Keine Schleichwerbung auf Mailinglisten -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHHuMz82U77t7MUWURAuB1AJ0enu3l33JALxgRYfB7MJ4btXYk6gCeKFTJ iAp4Ppq2fdOnnrqob2H92oI= =kuMn -END PGP SIGNATURE- ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
[jabberd2] Clustering
> Hi, > > I have a question about a way to implement of clustering in jabberd2. > Why don't we use software open source that run clusters? > It may be stupid but what's your opinion? > > Why do you developp another solutions made by others? > > http://openmosix.sourceforge.net/ > > There are another software if you search on google. > > > > ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
On 17 October 2007 at 2:20 PM, Tomasz Sterna wrote: > Dnia 17-10-2007, Sr o godzinie 13:52 -0700, Mark Doliner pisze: > > > When the router detects bind conflict. It tells the conflicting > > > components to "go to the lower level". > > > > And there would be a bind conflict when two sm instances try to > > connect to router with the id "jabber.org," right? > > And it would also happen when a single user logs in from two > > locations using two different resources? > > Exactly. > I've described it in "Another step:" part of the example. > > > > > > how does c2s decide which sm instance to forward the login to? > > > > > > C2S needs changes in the SM tracking logic. > > > When it receives advertisements of bare-jids, it starts routing by > > > bare-jids, not domain. Same for full-jids. > > > > When a user logs in they get assigned to one of the sm instances, right? > > Right. > The SM that responded to the session creation request. > We need to record it in the SM. :-) > > (Yes, we do crystallize the idea now. :-) ) > > > > How would c2s decide which sm instance they should be assigned to? > > Would the router make that decision somehow? > > Yes. It's router to decide which instance receives the packet addressed > to the SM (domain) itself. Round-robin comes to mind. Or we could count > sessions hanging on the SM and pick the less busy one. > > Using the routing trees (last paragraph) it's trivial to get needed > information. I've uploaded an initial patch for this to http://jabberd2.xiaoka.com/ticket/170 . It's not finished, but I am at least able to have two sm instances connect to one router, and have two clients log in and exchange messages. I wrote a more detailed description in the ticket. -Mark ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
Dnia 17-10-2007, Śr o godzinie 23:20 +0200, Tomasz Sterna pisze: > We need to record it in the SM. :-) In the C2S I meant. -- /\_./o__ Tomasz Sterna (/^/(_^^' Xiaoka.com ._.(_.)_ XMPP: [EMAIL PROTECTED] ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
Dnia 17-10-2007, Śr o godzinie 13:52 -0700, Mark Doliner pisze: > > When the router detects bind conflict. It tells the conflicting > > components to "go to the lower level". > > And there would be a bind conflict when two sm instances try to > connect to router with the id "jabber.org," right? > And it would also happen when a single user logs in from two > locations using two different resources? Exactly. I've described it in "Another step:" part of the example. > > > how does c2s decide which sm instance to forward the login to? > > > > C2S needs changes in the SM tracking logic. > > When it receives advertisements of bare-jids, it starts routing by > > bare-jids, not domain. Same for full-jids. > > When a user logs in they get assigned to one of the sm instances, right? Right. The SM that responded to the session creation request. We need to record it in the SM. :-) (Yes, we do crystallize the idea now. :-) ) > How would c2s decide which sm instance they should be assigned to? > Would the router make that decision somehow? Yes. It's router to decide which instance receives the packet addressed to the SM (domain) itself. Round-robin comes to mind. Or we could count sessions hanging on the SM and pick the less busy one. Using the routing trees (last paragraph) it's trivial to get needed information. -- /\_./o__ Tomasz Sterna (/^/(_^^' Xiaoka.com ._.(_.)_ XMPP: [EMAIL PROTECTED] ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
> -Original Message- > From: Tomasz Sterna > Sent: Wednesday, October 17, 2007 1:34 PM > To: jabberd2 development > Subject: Re: [jabberd2] Clustering > > Dnia 17-10-2007, Śr o godzinie 13:05 -0700, Mark Doliner pisze: > > Doesn't this make the router a bottleneck? > > Router has almost no logic - it only passes packets through. > I'm worried more about the fact that it is a single point of failure. > > We may introduce router mesh. Set of interconnected routers that > exchange their routing information. > A router receiving bound name advertisement records it in the routing > table. Same for routing method (domain/bare-jid/full-jid) change > requests when connected router detects conflict. > > Example: > sm1 - router1 - router2 - sm2 > > router1 table: > sm1 > router2 > sm2* - advertisement record (presence from component) > > router2 table: > sm2 1234 > router1 2345 > sm1* 2345 - advertisement record (presence from component) > > > > > Creating a clustering component [...] > > I don't like the idea of introducing yet another component in the > puzzle. The clustering functionality should be handled in the router. Hmm, it DOES make sense to have the router do the routing... And the router mesh idea seems cleaner than having a cluster component route packets from one cluster to another using the s2s component... > > Also, when would sm bind to a bare jid? > > When the router detects bind conflict. It tells the conflicting > components to "go to the lower level". And there would be a bind conflict when two sm instances try to connect to router with the id "jabber.org," right? And it would also happen when a single user logs in from two locations using two different resources? > > how does c2s decide which sm instance to forward the login to? > > C2S needs changes in the SM tracking logic. > When it receives advertisements of bare-jids, it starts routing by > bare-jids, not domain. Same for full-jids. When a user logs in they get assigned to one of the sm instances, right? How would c2s decide which sm instance they should be assigned to? Would the router make that decision somehow? -Mark ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
Dnia 17-10-2007, Śr o godzinie 13:05 -0700, Mark Doliner pisze: > Doesn't this make the router a bottleneck? Router has almost no logic - it only passes packets through. I'm worried more about the fact that it is a single point of failure. We may introduce router mesh. Set of interconnected routers that exchange their routing information. A router receiving bound name advertisement records it in the routing table. Same for routing method (domain/bare-jid/full-jid) change requests when connected router detects conflict. Example: sm1 - router1 - router2 - sm2 router1 table: sm1 router2 sm2* - advertisement record (presence from component) router2 table: sm2 1234 router1 2345 sm1* 2345 - advertisement record (presence from component) > Creating a clustering component [...] I don't like the idea of introducing yet another component in the puzzle. The clustering functionality should be handled in the router. > Also, when would sm bind to a bare jid? When the router detects bind conflict. It tells the conflicting components to "go to the lower level". > how does c2s decide which sm instance to forward the login to? C2S needs changes in the SM tracking logic. When it receives advertisements of bare-jids, it starts routing by bare-jids, not domain. Same for full-jids. -- /\_./o__ Tomasz Sterna (/^/(_^^' Xiaoka.com ._.(_.)_ XMPP: [EMAIL PROTECTED] ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
Doesn't this make the router a bottleneck? Creating a clustering component as specified at http://jabberd2.xiaoka.com/wiki/Clustering appeals to me because it allows for multiple routers in addition to just multiple sm's. It WOULD be nice to be able to take advantage of multiple processors in a single machine, though. Also, when would sm bind to a bare jid? Would it happen when the sm process is started? Or would it happen after the user authenticates? If it happens after authentication, how does c2s decide which sm instance to forward the login to? -Mark > -Original Message- > From: [EMAIL PROTECTED] [mailto:jabberd2- > [EMAIL PROTECTED] On Behalf Of Tomasz Sterna > Sent: Wednesday, October 17, 2007 11:49 AM > To: jabberd2 development > Subject: Re: [jabberd2] Clustering > > Dnia 17-10-2007, Śr o godzinie 10:34 -0700, Mark Doliner pisze: > > I'm interested in adding clustering support to jabberd2. > > Most of us are. :-) > > > We're noticing that sm is definitely a bottleneck in our jabberd2 > installation, > > Yes. SM is definitely a bottleneck for big deployments. > > > I have an idea how to do it a flexible and efficient way. > A way that would fix the SM problem too. > > I call it "dynamic routing tables". > The idea is similar to the way TCP network handles the network > segmentation. > > The router currently holds a list of connected components and its bound > names. It uses this list to make a decision where to put a packet coming > through. > > componentport > -- > c2s 1 > resolver 1 > example.com 1 > test.org 1 > test2.org1 > > So, when it handles packet destined to [EMAIL PROTECTED]/res it puts it > to 1 stream. (or falls back to defaultroute) > > One component is able to bind more than one name. > > But no more than one component can bind a name. Logically - router would > not know where to put the packet in. > > We need to change the way, this bind conflict is handled. > Instead of disallowing new bind of 'example.com' to a component, router > would need to signal to the requesting component and component on port > 1, that they need to switch to bare-jid binding from domain binding. > > Then 1 and 1 would request binds of all it's bare jid's and the > routing table would look: > > component port > --- > c2s 1 > resolver 1 > [EMAIL PROTECTED] 1 > [EMAIL PROTECTED] 1 > [EMAIL PROTECTED] 1 > test.org 1 > test2.org 1 > [EMAIL PROTECTED] 1 > [EMAIL PROTECTED] 1 > > This way, router receiving packet for [EMAIL PROTECTED]/res would put it > to 1 stream and for [EMAIL PROTECTED]/home would go to 1. > Packets to SM itself would go to one of the shared routes - for load > balancing. round-robin? > > Another step: component on port 1 requests bind of [EMAIL PROTECTED] > (user connected with another resource) generating conflict with > component on port 1. Router would then to signal to both components, > that they need to switch to full-jid binding from bare-jid binding. > > After components bind all full JIDs the routing table would look like: > > component port > --- > c2s1 > resolver 1 > [EMAIL PROTECTED]/cl1 > [EMAIL PROTECTED]/res 1 > [EMAIL PROTECTED]/res1 > [EMAIL PROTECTED]/res1 > test.org 1 > test2.org 1 > [EMAIL PROTECTED]/home 1 > [EMAIL PROTECTED]/work 1 > > And router can do a routing for [EMAIL PROTECTED]/res to 1 stream. > On this level we would need to track resource priorities too, but this > is a minor. > > This dynamically expanding router tables would allow us to plug many SM > processes handling same domain even on single host. That would > distribute load to many OS processes. > > Balancer code could be reused to implement more than one default route > S2S processes. Not only for incoming, but outgoing connections too. > (When more than one component requests default bind.) > > > Implementation note: > This routing table would rather be a routing tree with domains on first > level, bare-jids on the second and full jids on the third. > > -- > /\_./o__ Tomasz Sterna > (/^/(_^^' Xiaoka.com > ._.(_.)_ XMPP: [EMAIL PROTECTED] > > ___ > Jabberd2 mailing list > Jabberd2@lists.xiaoka.com > http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
Re: [jabberd2] Clustering
Dnia 17-10-2007, Śr o godzinie 10:34 -0700, Mark Doliner pisze: > I'm interested in adding clustering support to jabberd2. Most of us are. :-) > We're noticing that sm is definitely a bottleneck in our jabberd2 > installation, Yes. SM is definitely a bottleneck for big deployments. I have an idea how to do it a flexible and efficient way. A way that would fix the SM problem too. I call it "dynamic routing tables". The idea is similar to the way TCP network handles the network segmentation. The router currently holds a list of connected components and its bound names. It uses this list to make a decision where to put a packet coming through. componentport -- c2s 1 resolver 1 example.com 1 test.org 1 test2.org1 So, when it handles packet destined to [EMAIL PROTECTED]/res it puts it to 1 stream. (or falls back to defaultroute) One component is able to bind more than one name. But no more than one component can bind a name. Logically - router would not know where to put the packet in. We need to change the way, this bind conflict is handled. Instead of disallowing new bind of 'example.com' to a component, router would need to signal to the requesting component and component on port 1, that they need to switch to bare-jid binding from domain binding. Then 1 and 1 would request binds of all it's bare jid's and the routing table would look: component port --- c2s 1 resolver 1 [EMAIL PROTECTED] 1 [EMAIL PROTECTED] 1 [EMAIL PROTECTED] 1 test.org 1 test2.org 1 [EMAIL PROTECTED] 1 [EMAIL PROTECTED] 1 This way, router receiving packet for [EMAIL PROTECTED]/res would put it to 1 stream and for [EMAIL PROTECTED]/home would go to 1. Packets to SM itself would go to one of the shared routes - for load balancing. round-robin? Another step: component on port 1 requests bind of [EMAIL PROTECTED] (user connected with another resource) generating conflict with component on port 1. Router would then to signal to both components, that they need to switch to full-jid binding from bare-jid binding. After components bind all full JIDs the routing table would look like: component port --- c2s1 resolver 1 [EMAIL PROTECTED]/cl1 [EMAIL PROTECTED]/res 1 [EMAIL PROTECTED]/res1 [EMAIL PROTECTED]/res1 test.org 1 test2.org 1 [EMAIL PROTECTED]/home 1 [EMAIL PROTECTED]/work 1 And router can do a routing for [EMAIL PROTECTED]/res to 1 stream. On this level we would need to track resource priorities too, but this is a minor. This dynamically expanding router tables would allow us to plug many SM processes handling same domain even on single host. That would distribute load to many OS processes. Balancer code could be reused to implement more than one default route S2S processes. Not only for incoming, but outgoing connections too. (When more than one component requests default bind.) Implementation note: This routing table would rather be a routing tree with domains on first level, bare-jids on the second and full jids on the third. -- /\_./o__ Tomasz Sterna (/^/(_^^' Xiaoka.com ._.(_.)_ XMPP: [EMAIL PROTECTED] ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
[jabberd2] Clustering
Hi! I'm interested in adding clustering support to jabberd2. We're noticing that sm is definitely a bottleneck in our jabberd2 installation, and we think clustering would eliminate this. I've read through all the clustering pages on the wiki and think I have a pretty good idea of what needs to be done. Has anyone done any work on this? Or have any suggestions or advice? I saw that there's a patch from Bruce Campbell at http://jabberd2.xiaoka.com/ticket/25 for some minor changes to sm. -Mark ___ Jabberd2 mailing list Jabberd2@lists.xiaoka.com http://lists.xiaoka.com/listinfo.cgi/jabberd2-xiaoka.com
