Re: painful mupdate syncs between front-ends and database server

2009-10-31 Thread Bron Gondwana
On Fri, Oct 30, 2009 at 03:24:25PM -0400, Michael Bacon wrote: I haven't had the guts to roll the patched, CVS version into production as our primary mupdate server, but I did put it in on a test machine in replica mode. My measurement was on a clean server (no pre-existing mailboxes.db), and

Re: painful mupdate syncs between front-ends and database server

2009-10-31 Thread David Carter
On Fri, 30 Oct 2009, Michael Bacon wrote: On all systems in the murder, we'll see instances where the mupdate process goes into a spin where, in truss, it's an endless repeat of fcntl, stat, fstat, fcntl, thousands of times over. These execute extremely quickly, but I do wonder if we're

Re: painful mupdate syncs between front-ends and database server

2009-10-30 Thread Michael Bacon
I apologize for not responding sooner here. I've had my head down in the code and doing some tests, including playing with Bron's patch here. I haven't had the guts to roll the patched, CVS version into production as our primary mupdate server, but I did put it in on a test machine in replica

Re: painful mupdate syncs between front-ends and database server

2009-10-30 Thread Michael Bacon
--On October 19, 2009 9:37:57 PM -0400 Wesley Craig w...@umich.edu wrote: How are your frontend mupdate processes authenticating to your mupdate master? And what version of Kerberos are you using (anticipating the answer to your first question)? I suspect that you're getting a GSSAPI expired

Re: painful mupdate syncs between front-ends and database server

2009-10-30 Thread Michael Bacon
--On October 20, 2009 12:13:05 PM +0200 Cyril Servant elfejoy...@gmail.com wrote: Hello, Here we had a similar situation : more than a million mailboxes, and each MUPDATE sync was very long (when it succeeded). Now, we bypass the problem : we get rid of the MUPDATE (and the skiplist

Re: painful mupdate syncs between front-ends and database server

2009-10-20 Thread Cyril Servant
Hello, 2009/10/19 Michael Bacon bac...@email.unc.edu: Hello, list, Today we're enjoying our first full work day of independence from the old monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU boards since then, but that's it), and on our new shiny cluster of T5220's

painful mupdate syncs between front-ends and database server

2009-10-19 Thread Michael Bacon
Hello, list, Today we're enjoying our first full work day of independence from the old monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU boards since then, but that's it), and on our new shiny cluster of T5220's that are mostly happily operating as a murder. I say mostly

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Dan White
On 19/10/09 16:38 -0400, Michael Bacon wrote: I say mostly because while most of the times the thing handles our 80,000 users and 14,000+ simultaneous connections like a champ, some of the time, we get some extreme pain, mostly due to syncs between the MUPDATE master and the front-end servers.

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Andrew Morgan
On Mon, 19 Oct 2009, Michael Bacon wrote: Hello, list, Today we're enjoying our first full work day of independence from the old monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU boards since then, but that's it), and on our new shiny cluster of T5220's that are mostly

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Michael Bacon
--On October 19, 2009 2:13:03 PM -0700 Andrew Morgan mor...@orst.edu wrote: What is causing a (re)sync of the frontends? Normally this should only happen when you start Cyrus on a frontend, right? I am not entirely sure. I think what may be happening is that the slave mupdate requests get

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Brian Awood
On Monday 19 October 2009 @ 16:38, Michael Bacon wrote: I say mostly because while most of the times the thing handles our 80,000 users and 14,000+ simultaneous connections like a champ, some of the time, we get some extreme pain, mostly due to syncs between the MUPDATE master and the

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Andrew Morgan
On Mon, 19 Oct 2009, Michael Bacon wrote: --On October 19, 2009 2:13:03 PM -0700 Andrew Morgan mor...@orst.edu wrote: What is causing a (re)sync of the frontends? Normally this should only happen when you start Cyrus on a frontend, right? I am not entirely sure. I think what may be

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Wesley Craig
On 19 Oct 2009, at 17:37, Michael Bacon wrote: --On October 19, 2009 2:13:03 PM -0700 Andrew Morgan mor...@orst.edu wrote: What is causing a (re)sync of the frontends? Normally this should only happen when you start Cyrus on a frontend, right? I am not entirely sure. I think what may be

Re: painful mupdate syncs between front-ends and database server

2009-10-19 Thread Bron Gondwana
On Mon, 19 Oct 2009 16:38 -0400, Michael Bacon bac...@email.unc.edu wrote: When we spec'ed out our servers, we didn't put much I/O capacity into the front-end servers -- just a pair of mirrored 10k disks doing the OS, the logging, the mailboxes.db, and all the webmail action going on in