Hi Bill/all, I think in all fairness in regards to load balancing XMPP traffic, most of the strategies to date are custom-built and not formally spec'ed out or officially documented (if I'm in error, somone please point it out). Mickael's company does such custom solutions for enterprise-level IM apps.
A roadblock I'm running into in getting local telecomm carriers and cable providers (in Guam) to warm up to XMPP systems is getting them to stop thinking of network management in terms of web servers. Push/presence/rosters/PubSub is a completely different animal than http, as you effectively pointed out. The challenges are different, so I'm with you that in lieu of a model for load balancing we get some case studies. Blaine from Twitter's told me that clustering should happen at around the 5,000-user level in rosters, at the latest. I also foresee the web community crying out for BOSH, as they become more and more aware of it, to include some sort of XMPP variant of a web caching facility, if applicable. Good point made about this here: http://mail.jabber.org/pipermail/bosh/2008-July/000013.html This further proves your point about a different platform with different mechanics to achieve a common goal. Jas Sent from my BlackBerry® wireless device -----Original Message----- From: Bill de hOra <[email protected]> Date: Thu, 26 Feb 2009 09:55:31 To: XMPP and Social Networking,Two Great Tastes That Taste Great Together!<[email protected]> Subject: Re: [Social] Facebook and XMPP? Mickael Remond wrote: > Hello Bill, > > You wrote: > >> - load balancing. Anyone here know how to build out an XMPP >> deployment that has hundreds of thousands to low millions of users, >> never mind 50M+? > > Yes, we know several of them. Great, so where is that knowledge held within the XMPP community? The Web community's knowledge on how to scale is easily discovered. That doesn't mean to say it's easy or cheap to scale, just that the information on how to do so is widely available. Can I say the same for XMPP? I don't think so, hence my claim it's partly Voodoo today. So I still think for the most part people don't know how to deploy, and so resort to systems approaches they do understand, such as Comet/BOSH gateways. >> - clustering. Fwiw, I don't think it's easy enough to cluster XMPP >> servers compared to web farms. Generic Tools like pound/perlbal mostly >> don't exist in the XMPP world. The OSS stacks I've seen, ejabberd, >> openfire, jabberd2 are limited to what I'd call "lan scale". To work >> with long lived connections at this scale you have to be using epoll. > > XMPP can be improved. Some improvements will need to be made in protocol > other in implementation. You have a large progress margin to web scale. I'm specifically talking about the stacks though. > However, XMPP is definitely not "lan scale". I agree, here's what I said: "I'm talking about *deployments* here - I'm not questioning whether XMPP itself is a scalable protocol." > My last comment is that the problem you have to deal with is inherent to > the functional requirement in most case. XMPP is only a transport in > most case. If the requirements of the application is to distribute > millions of copy of a presence packet for example, you have to > distribute them no matter the technology you use. I don't agree with the transport point - XMPP works at a higher level on the stack. I think in many cases the requirement is to distribute lots of packets to a lot of people. Distributing the same packet to a lot of people sounds like pub/sub, which I'm fairly sure remains a challenge at the scales we're talking about. The problem with using HTTP as a transport for XMPP is that it might hurt XMPP - the history of SOAP/WS suggests treating HTTP as a transport doesn't work out. Eventually RESTful messaging will be built (probably via adding constraints) subsuming other messaging protocols (I'm also thinking about stuff like amqp). Ideally for me, XMPP/5222/5269 becomes as ubiquitous as HTTP/80. Btw - I'm not completely sold on the meme you have to integrate with browsers just because http/80 is ubiquitous. Or saying a major player needs to get behind XMPP. I wouldn't go that way - I would focus instead on user experience - does anyone here really think websites are a good basis for lifestream/activities UIs? To me, it's a) clear that web UIs aren't a great fit for quickly updating data, b) all the "social sites" user interfaces are converging towards messaging. Yet people are still doing things over http into browsers, arguably neither of which are a natural fit. I think XMPP can support a better kind of user experience for lifestream/activities style applications. Bill
