Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
--On 13. Juli 2009 15:36:07 +0200 Gabor Gombas wrote: On Mon, Jul 13, 2009 at 02:09:40PM +0200, Sebastian Hagedorn wrote: > 4 GB limit of 32 bit binaries? Perhaps, although I haven't seen it. That's only 3GB by default, 1GB of address space is reserved for the kernel. Also, the stack, the executable, and all the shared libraries the executable uses also occupy some address space. Good to know. Of course it's possible that it then tried to allocate one huge chunk, but I can't see that. Are there better tools to monitor the memory allocation of a process? strace -e trace=brk,mmap,munmap (well, this actually traces glibc's memory management, but it should show the failure). You. I had to specify mmap2, but then I see this: mmap2(NULL, 267374592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2218b000 mmap2(NULL, 858675, PROT_READ, MAP_SHARED, 43, 0) = 0x220b9000 munmap(0x220b9000, 858675) = 0 munmap(0x2218b000, 267374592) = 0 mmap2(NULL, 682827776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) brk(0) = 0x993d000 brk(0x32483000) = 0x993d000 mmap2(NULL, 682962944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x337c1000 munmap(0x337c1000, 258048) = 0 munmap(0x3390, 790528) = 0 mmap2(NULL, 682827776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) fatal error: Virtual memory exhausted So it really tries to allocate a rather large chunk! I guess that's one more reason to switch to 64-bit when we make he move to RHEL 5. Thanks! -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. pgp0WmEua5sro.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
On Mon, Jul 13, 2009 at 02:09:40PM +0200, Sebastian Hagedorn wrote: > >4 GB limit of 32 bit binaries? > > Perhaps, although I haven't seen it. That's only 3GB by default, 1GB of address space is reserved for the kernel. Also, the stack, the executable, and all the shared libraries the executable uses also occupy some address space. > Of course it's possible that it then tried to allocate one huge > chunk, but I can't see that. Are there better tools to monitor the > memory allocation of a process? strace -e trace=brk,mmap,munmap (well, this actually traces glibc's memory management, but it should show the failure). Gabor -- - MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences - Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
--On 13. Juli 2009 14:53:31 +0200 Pascal Gienger
wrote:
Sebastian Hagedorn schrieb:
fatal error: Virtual memory exhausted
Of course it's possible that it then tried to allocate one huge chunk,
but I can't see that. Are there better tools to monitor the memory
allocation of a process?
Swap file/partition full?
Unlikely.
Background:
I think the message "Virtual memory exhausted" is coming from your
operating system and not from the squatter process.
I disagree.
Squatter would have been said
switch (err) {
case SQUAT_ERR_OUT_OF_MEMORY:
fprintf(stderr, "SQUAT: Out of memory (%s)\n", s);
break;
But:
lib/xmalloc.c:fatal("Virtual memory exhausted", EC_TEMPFAIL);
This code actually can't be reached:
b->buf = (char*)xrealloc(b->buf, len);
if (b->buf == NULL) {
squat_set_last_error(SQUAT_ERR_OUT_OF_MEMORY);
return NULL;
}
If the xrealloc fails, the "fatal" above is called within that routine.
--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
.:.Regionales Rechenzentrum (RRZK).:.
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.
pgpUnC6euB2si.pgp
Description: PGP signature
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
Sebastian Hagedorn schrieb:
>>> fatal error: Virtual memory exhausted
> Of course it's possible that it then tried to allocate one huge chunk,
> but I can't see that. Are there better tools to monitor the memory
> allocation of a process?
Swap file/partition full?
Background:
I think the message "Virtual memory exhausted" is coming from your
operating system and not from the squatter process.
Squatter would have been said
switch (err) {
case SQUAT_ERR_OUT_OF_MEMORY:
fprintf(stderr, "SQUAT: Out of memory (%s)\n", s);
break;
So I think it is a Virtual Memory/Swap problem in your OS.
--
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
--On 13. Juli 2009 13:20:42 +0200 Pascal Gienger wrote: Sebastian Hagedorn schrieb: Processing index character 101, 681642 total words, temp file size is 2107147 fatal error: Virtual memory exhausted 4 GB limit of 32 bit binaries? Perhaps, although I haven't seen it. How much RAM does squatter allocate before it dies? I monitored the process like this: # while true; do grep VmSize /proc/16815/status; sleep 1; done; The last lines are: VmSize: 2453772 kB VmSize: 2454608 kB VmSize: 2454608 kB VmSize: 2454884 kB VmSize: 2192664 kB VmSize: 2192664 kB Of course it's possible that it then tried to allocate one huge chunk, but I can't see that. Are there better tools to monitor the memory allocation of a process? Thanks! -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. pgpGwJtb8GPrW.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
Sebastian Hagedorn schrieb:
> Processing index character 101, 681642 total words, temp file size is
> 2107147
> fatal error: Virtual memory exhausted
4 GB limit of 32 bit binaries?
How much RAM does squatter allocate before it dies?
--
Pascal Gienger
University of Konstanz, IT Services Department ("Rechenzentrum")
Electronic Communications and Web Services
Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
squatter exits with "fatal error: Virtual memory exhausted" on huge mailbox
Hi, we run 2.3.14 and we have one user with an enormous mailbox (not his INBOX!). It contains around 230,000 messages. We run squatter with a cronjob on all mailboxes. Recently this one mailbox has apparently become too big for squatter. When I try to squat it in verbose mode, it ends like this: ... Skipping tiny document part 'b563140' (size 0) Opening document part 's563140' Processing index character 7, 5 total words, temp file size is 35 Processing index character 10, 152 total words, temp file size is 1064 Processing index character 13, 152 total words, temp file size is 1064 Processing index character 15, 3 total words, temp file size is 21 Processing index character 23, 5 total words, temp file size is 35 Processing index character 27, 5 total words, temp file size is 35 Processing index character 30, 3 total words, temp file size is 21 Processing index character 32, 12938 total words, temp file size is 66346 Processing index character 33, 2692 total words, temp file size is 11856 Processing index character 39, 12553 total words, temp file size is 57119 Processing index character 40, 60791 total words, temp file size is 203723 Processing index character 41, 57972 total words, temp file size is 195005 Processing index character 42, 20959 total words, temp file size is 68807 Processing index character 43, 51335 total words, temp file size is 172871 Processing index character 44, 78879 total words, temp file size is 270078 Processing index character 45, 183196 total words, temp file size is 588133 Processing index character 46, 233839 total words, temp file size is 756846 Processing index character 47, 56284 total words, temp file size is 195995 Processing index character 48, 226012 total words, temp file size is 701263 Processing index character 49, 168692 total words, temp file size is 530284 Processing index character 50, 153898 total words, temp file size is 487575 Processing index character 51, 122957 total words, temp file size is 391515 Processing index character 52, 100478 total words, temp file size is 322795 Processing index character 53, 81760 total words, temp file size is 265813 Processing index character 54, 78645 total words, temp file size is 258465 Processing index character 55, 83934 total words, temp file size is 272204 Processing index character 56, 82022 total words, temp file size is 266903 Processing index character 57, 76410 total words, temp file size is 250698 Processing index character 58, 135314 total words, temp file size is 437180 Processing index character 59, 44962 total words, temp file size is 151528 Processing index character 60, 49223 total words, temp file size is 197205 Processing index character 61, 39112 total words, temp file size is 133934 Processing index character 62, 110296 total words, temp file size is 363204 Processing index character 63, 3887 total words, temp file size is 18737 Processing index character 64, 60070 total words, temp file size is 233407 Processing index character 88, 238 total words, temp file size is 770 Processing index character 91, 22210 total words, temp file size is 7 Processing index character 92, 1490 total words, temp file size is 6242 Processing index character 93, 17768 total words, temp file size is 75520 Processing index character 94, 226 total words, temp file size is 962 Processing index character 95, 54059 total words, temp file size is 179583 Processing index character 96, 106 total words, temp file size is 506 Processing index character 97, 389316 total words, temp file size is 1227616 Processing index character 98, 140247 total words, temp file size is 457214 Processing index character 99, 243003 total words, temp file size is 782841 Processing index character 100, 262183 total words, temp file size is 833170 Processing index character 101, 681642 total words, temp file size is 2107147 fatal error: Virtual memory exhausted We will try to get this user to split this mailbox or to clean it up, but I wondered if there is something else we can do. The machine has ample amounts of memory, so I'm unsure why the call to malloc would fail ... -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. pgpP2d9eLBSG3.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: ctl_mboxlist virtual memory exhausted !
thanks for your answer ! * the previous "large installation" we have is 30Millions users with lots of backends and store (cyrus + netapp filer ) , and an internaly developped pop3 proxy (usage was only pop3 at this time ;-) (we also have an imap proxy for a fraction of users), at the time we build it, things like nginx was not there ;-) * this one was "low usage", but large number of users, so we thought only one backend was needed, with the ability to move users one more backend if load dictate it. So we tought murder architecture was the way to go ... seems to be a bad idea ? note : storage is Hitachi HDS SAN on RAID5 volumes, replicated across 2 datacenters. * and yes, I applied the patches last week, and checking now if there is no more "bugs". note : I also got some "hangs" with berkeley DB (4.3.27) : #4 0xb7d89edd in ___newselect_nocancel () from /lib/libc.so.6 #5 0x080e4a08 in __os_sleep_cyrus_imapd_db (dbenv=0x0, secs=1, usecs=0) at ../dist/../os/os_sleep.c:84 #6 0x080e4b36 in __os_yield_cyrus_imapd_db (dbenv=0x0, usecs=100) at ../dist/../os/os_spin.c:110 #7 0x080f0927 in __db_fcntl_mutex_lock_cyrus_imapd_db (dbenv=0x823dac0, mutexp=0xb7721360) at ../dist/../mutex/mut_fcntl.c:95 #8 0x080c5eb4 in __lock_get_internal_cyrus_imapd_db (lt=0x823ddb8, locker=2149230760, flags=0, obj=0x823fecc, lock_mode=DB_LOCK_WRITE, timeout=0, lock=0xbfcc4bf0) at ../dist/../lock/lock.c:820 #9 0x080c45b8 in __lock_vec_cyrus_imapd_db (dbenv=0x823dac0, locker=2149230760, flags=0, list=0xbfcc4be0, nlist=2, -Message d'origine- De : Bron Gondwana [mailto:[EMAIL PROTECTED] Envoyé : mardi 25 mars 2008 23:28 À : Brasseur Valery; info-cyrus@lists.andrew.cmu.edu Objet : Re: ctl_mboxlist virtual memory exhausted ! On Tue, 25 Mar 2008 09:31:40 +0100, "Brasseur Valery" <[EMAIL PROTECTED]> said: > Hi, > > I am running 2.3.11, with 2 Millions users (4M mailboxes ;-) > > when trying to do a ctl_mboxlist -m, after some time (a few second !) I > got a "virtual memory exhausted", and i can see that the process is > allocating more than 3Gb of memory ! Ouch. That hurts > did some of you encontered this ? > any way to bypass ? We split our Cyrus instance up into 300Gb data partitions. We currently have 56 stores (112 partitions thanks to replication). Obviously you need infrastructure to manage this, and some form of frontend proxy to direct user logins to the correct store (we use nginx). Further, any users who need to share mailboxes must be on the same store. Still, things are a lot faster when your average mailboxes.db is only 4.5Mb in size (having just checked the one for the store my mailbox is on) > I also got a lot's of skiplist corruption when file size is around 700Mb > for mailboxes.db, and mupdate process getting 100% of CPU when it's > arrive !!! > any ideas are welcome ! Are you using: http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-safelock-2.3.11.diff http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-state-2.3.11.diff http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-transactions-2.3.10.diff Finally, are you running a 32 bit operating system? With a 700Mb mailboxes.db being mmaped, you might be pushing close to the available process memory space. Running a 64 bit kernel would probably help a lot there (you will of course need to have 64 bit hardware!) I would seriously recommend against having 2 million users all on one machine on disaster recovery principles - it takes far too long to copy that much data onto modern drives, so if you lose your drive unit then getting users back up and running looks like about a week's sitting there watching data copy. Yes, I have done that before. That's why we run partitions that can rebuild from scratch in 6 hours now. Bron. -- Bron Gondwana [EMAIL PROTECTED] Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be trig
Re: ctl_mboxlist virtual memory exhausted !
On Tue, 25 Mar 2008 09:31:40 +0100, "Brasseur Valery" <[EMAIL PROTECTED]> said: > Hi, > > I am running 2.3.11, with 2 Millions users (4M mailboxes ;-) > > when trying to do a ctl_mboxlist -m, after some time (a few second !) I > got a "virtual memory exhausted", and i can see that the process is > allocating more than 3Gb of memory ! Ouch. That hurts > did some of you encontered this ? > any way to bypass ? We split our Cyrus instance up into 300Gb data partitions. We currently have 56 stores (112 partitions thanks to replication). Obviously you need infrastructure to manage this, and some form of frontend proxy to direct user logins to the correct store (we use nginx). Further, any users who need to share mailboxes must be on the same store. Still, things are a lot faster when your average mailboxes.db is only 4.5Mb in size (having just checked the one for the store my mailbox is on) > I also got a lot's of skiplist corruption when file size is around 700Mb > for mailboxes.db, and mupdate process getting 100% of CPU when it's > arrive !!! > any ideas are welcome ! Are you using: http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-safelock-2.3.11.diff http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-state-2.3.11.diff http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-transactions-2.3.10.diff Finally, are you running a 32 bit operating system? With a 700Mb mailboxes.db being mmaped, you might be pushing close to the available process memory space. Running a 64 bit kernel would probably help a lot there (you will of course need to have 64 bit hardware!) I would seriously recommend against having 2 million users all on one machine on disaster recovery principles - it takes far too long to copy that much data onto modern drives, so if you lose your drive unit then getting users back up and running looks like about a week's sitting there watching data copy. Yes, I have done that before. That's why we run partitions that can rebuild from scratch in 6 hours now. Bron. -- Bron Gondwana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
ctl_mboxlist virtual memory exhausted !
Hi, I am running 2.3.11, with 2 Millions users (4M mailboxes ;-) when trying to do a ctl_mboxlist -m, after some time (a few second !) I got a "virtual memory exhausted", and i can see that the process is allocating more than 3Gb of memory ! did some of you encontered this ? any way to bypass ? I also got a lot's of skiplist corruption when file size is around 700Mb for mailboxes.db, and mupdate process getting 100% of CPU when it's arrive !!! any ideas are welcome ! thanks Ce message et les pi?ces jointes sont confidentiels et r?serv?s ? l'usage exclusif de ses destinataires. Il peut ?galement ?tre prot?g? par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir imm?diatement l'exp?diteur et de le d?truire. L'int?grit? du message ne pouvant ?tre assur?e sur Internet, la responsabilit? du groupe Atos Origin ne pourra ?tre recherch?e quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'exp?diteur ne donne aucune garantie ? cet ?gard et sa responsabilit? ne saurait ?tre recherch?e pour tout dommage r?sultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: syncserver[4715]: Fatal error: Virtual memory exhausted
Yeah, that's a good point. I use sync_batch_size of 100,000. Regarding the original question, there are other operations that will exhaust memory as well. We've had two recent reports of Thunderbird users who unexpectedly developed extremely large cyrus.* files in their trash. Since cyrus.* files get mmap'd, a very large one may exhaust the address space of sync_server (or imapd for that matter). :wes On 18 Apr 2007, at 07:16, Bron Gondwana wrote: If you're using 2.3.8 just add: sync_batch_size: 1000 to your imapd.conf (or any other value - this works fine for me) The above patch is probably fine too, but I still prefer to stop the unbounded edge case regardless. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: syncserver[4715]: Fatal error: Virtual memory exhausted
On 18 Apr 2007, at 01:27, Simon Matter wrote: I'm still wondering why the code is there, can anybody comment on this? The buffer in question is dynamically sized. I gather that an earlier version of the code pre-allocated 5 paths. A later version allowed an arbitrary number, adjusting the buffer size appropriately. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: syncserver[4715]: Fatal error: Virtual memory exhausted
On Tue, Apr 17, 2007 at 04:28:04PM -0400, Wesley Craig wrote: > You probably encountered a single very large mailbox. This patch: > > --- cyrus-imapd-2.3.8/imap/sync_support.c 2006-11-30 > 12:11:20.0 -0500 > +++ cyrus-imapd-2.3.8p3/imap/sync_support.c 2007-04-12 > 13:27:49.0 -0400 > @@ -914,9 +914,9 @@ > result = xzmalloc(sizeof(struct sync_message)); > message_uuid_set_null(&result->uuid); > > -result->msg_path = xzmalloc(5 * (MAX_MAILBOX_PATH+1) * sizeof > (char)); > +result->msg_path = xzmalloc((MAX_MAILBOX_PATH+1) * sizeof(char)); > result->msg_path_end = result->msg_path + > - 5 * (MAX_MAILBOX_PATH+1) * sizeof(char); > + (MAX_MAILBOX_PATH+1) * sizeof(char); > > snprintf(result->stagename, sizeof(result->stagename), "%lu.", > l->count); > > Reduces the space that sync_server allocates when syncing a mailbox. > The "5 *" doesn't really do anything useful, other than waste a lot > of space. That's only a factor of 5 fix. The real problem is that there's an unbounded stream size that will be fed into sync_server. I have a patch that limits it to 1000, and it's been accepted upstream since. If you're using 2.3.8 just add: sync_batch_size: 1000 to your imapd.conf (or any other value - this works fine for me) The above patch is probably fine too, but I still prefer to stop the unbounded edge case regardless. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: syncserver[4715]: Fatal error: Virtual memory exhausted
> You probably encountered a single very large mailbox. This patch: > > --- cyrus-imapd-2.3.8/imap/sync_support.c 2006-11-30 > 12:11:20.0 -0500 > +++ cyrus-imapd-2.3.8p3/imap/sync_support.c 2007-04-12 > 13:27:49.0 -0400 > @@ -914,9 +914,9 @@ > result = xzmalloc(sizeof(struct sync_message)); > message_uuid_set_null(&result->uuid); > > -result->msg_path = xzmalloc(5 * (MAX_MAILBOX_PATH+1) * sizeof > (char)); > +result->msg_path = xzmalloc((MAX_MAILBOX_PATH+1) * sizeof(char)); > result->msg_path_end = result->msg_path + > - 5 * (MAX_MAILBOX_PATH+1) * sizeof(char); > + (MAX_MAILBOX_PATH+1) * sizeof(char); > > snprintf(result->stagename, sizeof(result->stagename), "%lu.", > l->count); I'm still wondering why the code is there, can anybody comment on this? Simon > > Reduces the space that sync_server allocates when syncing a mailbox. > The "5 *" doesn't really do anything useful, other than waste a lot > of space. > > On 17 Apr 2007, at 15:24, Per olof Ljungmark wrote: >> After performing an initial sync of 65G worth of mailboxes the >> above error occurred. I kept an eye on the process initally but saw >> nothing worrying, then it died about halfway through the sync. >> >> Could I have hit some system limits? Vanilla FreeBSD 6.2. >> I'm starting off fresh now again to try to figure it out. >> >> And, while at it, here are one more sync related: >> * The "sync_shutdown_file:" switch, does it mean that the presence >> of this file will stop the sync process? At least that is how I >> intepret it... > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: syncserver[4715]: Fatal error: Virtual memory exhausted
Wesley Craig wrote: You probably encountered a single very large mailbox. This patch: Thanks for the patch, applied it, rebuilt and wiped out test replica config and spool. One puzzling thing so far: At first try, rolling replication started "by itself", replicating the complete spool. At this eecond go, this does not occur. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: syncserver[4715]: Fatal error: Virtual memory exhausted
You probably encountered a single very large mailbox. This patch: --- cyrus-imapd-2.3.8/imap/sync_support.c 2006-11-30 12:11:20.0 -0500 +++ cyrus-imapd-2.3.8p3/imap/sync_support.c 2007-04-12 13:27:49.0 -0400 @@ -914,9 +914,9 @@ result = xzmalloc(sizeof(struct sync_message)); message_uuid_set_null(&result->uuid); -result->msg_path = xzmalloc(5 * (MAX_MAILBOX_PATH+1) * sizeof (char)); +result->msg_path = xzmalloc((MAX_MAILBOX_PATH+1) * sizeof(char)); result->msg_path_end = result->msg_path + - 5 * (MAX_MAILBOX_PATH+1) * sizeof(char); + (MAX_MAILBOX_PATH+1) * sizeof(char); snprintf(result->stagename, sizeof(result->stagename), "%lu.", l->count); Reduces the space that sync_server allocates when syncing a mailbox. The "5 *" doesn't really do anything useful, other than waste a lot of space. On 17 Apr 2007, at 15:24, Per olof Ljungmark wrote: After performing an initial sync of 65G worth of mailboxes the above error occurred. I kept an eye on the process initally but saw nothing worrying, then it died about halfway through the sync. Could I have hit some system limits? Vanilla FreeBSD 6.2. I'm starting off fresh now again to try to figure it out. And, while at it, here are one more sync related: * The "sync_shutdown_file:" switch, does it mean that the presence of this file will stop the sync process? At least that is how I intepret it... Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
syncserver[4715]: Fatal error: Virtual memory exhausted
After performing an initial sync of 65G worth of mailboxes the above error occurred. I kept an eye on the process initally but saw nothing worrying, then it died about halfway through the sync. Could I have hit some system limits? Vanilla FreeBSD 6.2. I'm starting off fresh now again to try to figure it out. And, while at it, here are one more sync related: * The "sync_shutdown_file:" switch, does it mean that the presence of this file will stop the sync process? At least that is how I intepret it... Thanks again... Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Virtual memory exhausted
Hi All.
I am using cyrus-imapd 2.2.12 with nntp-functions enabled.
I was using instructions from
http://asg.web.cmu.edu/cyrus/download/imapd/install-netnews.html to get
some nntp-groups to my local server and to be able to access all the
posts history using IMAP
It went fine, until I tried to get a really big newsgroup..
looking thru /var/spool/imap/folder/corresponding/to/group I can see
that I managed to get 12708 messages.. but the problem is, that I just
can't access those using IMAP-protocol.
Here is the listing of the session:
* OK t1.cjlohbi.su Cyrus IMAP4 v2.2.12 server ready
1 login myusername mypassword
1 OK User logged in
2 id nil
* ID ("name" "Cyrus IMAPD" "version" "v2.2.12 2005/02/14 16:43:51"
"vendor" "Project Cyrus" "support-url" "http://asg.web.cmu.edu/cyrus";
"os" "OpenBSD" "os-version" "3.6" "environment" "Built w/Cyrus SASL
2.1.13; Running w/Cyrus SASL 2.1.13; Built w/Sleepycat Software:
Berkeley DB 4.2.52: (December 3, 2003); Running w/Sleepycat Software:
Berkeley DB 4.2.52: (December 3, 2003); Built w/OpenSSL 0.9.7d 17 Mar
2004; Running w/OpenSSL 0.9.7d 17 Mar 2004; CMU Sieve 2.2; mmap =
nommap; lock = fcntl; nonblock = fcntl; auth = unix; idle = idled")
2 OK Completed
3 select nntp.name.of.that.newsgroup
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Seen)]
* BYE Fatal error: Virtual memory exhausted
Connection closed by foreign host.
Everything works fine with smaller groups.
Also, I was watching top, during this session - I didn't see a single
sign of memory exhaustion - there's plenty of it.
and, yes, there are no ulimits set
should it be considered a bug report?
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: ctl_mboxlist : Virtual memory exhausted
On Fri, 2004-07-16 at 22:40, andrey wrote: > When doing ctl_mboxlist -m we see: Fatal error: Virtual memory exhausted > The size of mailboxes.db is 57 Mb. If anyone know the reason and the way of solving > this problem? > Thanks. OS FreeBSD 4.10 Perhaps check your ulimit to see what the maximim virtual memory size is, and try increasing it. I don't know if that's it, but it could be. -- Craig Ringer --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
ctl_mboxlist : Virtual memory exhausted
When doing ctl_mboxlist -m we see: Fatal error: Virtual memory exhausted The size of mailboxes.db is 57 Mb. If anyone know the reason and the way of solving this problem? Thanks. OS FreeBSD 4.10 Andrey Semakin
ctl_mboxlist: Virtual memory exhausted
Good day. When doing ctl_mboxlist -m we see: Fatal error: Virtual memory exhausted The size of mailboxes.db is 57 Mb. If anyone know the reason and the way of solving this problem? Thanks. Andrey Semakin
Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(
On Wed, 2 Jul 2003, Pascal Gienger wrote:
> No it is not a DB problem, it IS a sieve problem:
>
> Look at this piece of sieve code:
>
> if header :contains "From" "Muchsel" {
> addflag "\\Flagged";
> fileinto "INBOX";
> removeflag "\\Flagged";
> }
>
>
> It is generated by Horde Ingo. lmtpd breaks with "4.3.0 virtual memory
> exhausted". Ok the "removeflag" is worthless because the message is already
> in the INBOX - but it should be silently ignored.
>
> When removing the "removeflag"-Statement, lmtpd will accept the mail
> normally.
Hmmm, I can't duplicate this. I've tried:
if header :contains "From" "rjs3" {
addflag "\\Deleted";
fileinto "INBOX";
removeflag "\\Deleted";
}
which behaves as I expect.
-Rob
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456
Research Systems Programmer * /usr/contributed Gatekeeper
Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(
John Alton Tamplin <[EMAIL PROTECTED]> wrote:
Pascal Gienger wrote:
What version of DB are you running? Does this happen immediately or only
after it has been running a while? On Solaris 9 with DB 4.1.24-25, we
had a problem where DB functions would eventually return ENOSPC, which
No it is not a DB problem, it IS a sieve problem:
Look at this piece of sieve code:
if header :contains "From" "Muchsel" {
addflag "\\Flagged";
fileinto "INBOX";
removeflag "\\Flagged";
}
It is generated by Horde Ingo. lmtpd breaks with "4.3.0 virtual memory
exhausted". Ok the "removeflag" is worthless because the message is already
in the INBOX - but it should be silently ignored.
When removing the "removeflag"-Statement, lmtpd will accept the mail
normally.
Pascal
--
Pascal Gienger, Kanzleistrasse 14, D-78462 Konstanz
Postfix+SASL-AUTH+LDAP Solaris 8: http://pgienger.de/postfix/
Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(
Pascal Gienger wrote: I am using Cyrus IMAP 2.1.12 (Suse Linux package, no source install allowed here), and we see some errors from lmtpd saying "Virtual memory exhausted". ulimits are unlimited for every resource (no limit), and there are 4 Gigabytes of swap used by 5% and 2 Gigabytes of RAM. System is GNU/Linux 2.4.20-64GB. Our postfix mailer tries to deliver a message to a local lmtpd (Cyrus system) and here is the result: Jun 30 07:38:43 rauch postfix/lmtp[3129]: EC7FD4400088: to=<[EMAIL PROTECTED]>, orig_to=<[EMAIL PROTECTED]>, relay=/mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp], delay=142730, status=deferred (host /mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp] said: 421 4.3.0 lmtpd: Virtual memory exhausted (in reply to end of DATA command)) Has anybody seen this already? Which memory lmtpd claims to be exhausted? With no process limits at all? What version of DB are you running? Does this happen immediately or only after it has been running a while? On Solaris 9 with DB 4.1.24-25, we had a problem where DB functions would eventually return ENOSPC, which might be interpreted as "Virtual memory exhausted". There has been a patch posted to Cyrus to fix this problem, but if you can't use a source package you might have to downgrade to DB 4.0.14 if this is your problem. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: 4.3.0 lmtpd: Virtual memory exhausted - problem :(
I investigated and found a curiosity:
That user has installed a sieve script to sort mail.
In his script he used the keyword
keep;
to keep the mail in his INBOX.
This "keep" is going havoc ("virtual memory exhausted").
After replacing "keep" with 'fileinto "INBOX"' all things went
well.
Is this a bug in the sieve function of lmtpd?
here the NON WORKING part:
if header :comparator "i;ascii-casemap" :contains "From" "Muchsel" {
addflag "\\Flagged";
keep;
}
This sieve code results in "4.3.0 lmtpd: Virtual memory exhausted".
here the WORKING variant:
if header :comparator "i;ascii-casemap" :contains "From" "Muchsel" {
addflag "\\Flagged";
fileinto "INBOX";
}
Any comments?
Pascal
4.3.0 lmtpd: Virtual memory exhausted - problem :(
I am using Cyrus IMAP 2.1.12 (Suse Linux package, no source install allowed here), and we see some errors from lmtpd saying "Virtual memory exhausted". ulimits are unlimited for every resource (no limit), and there are 4 Gigabytes of swap used by 5% and 2 Gigabytes of RAM. System is GNU/Linux 2.4.20-64GB. Our postfix mailer tries to deliver a message to a local lmtpd (Cyrus system) and here is the result: Jun 30 07:38:43 rauch postfix/lmtp[3129]: EC7FD4400088: to=<[EMAIL PROTECTED]>, orig_to=<[EMAIL PROTECTED]>, relay=/mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp], delay=142730, status=deferred (host /mail/cyrus/socket/lmtp[/mail/cyrus/socket/lmtp] said: 421 4.3.0 lmtpd: Virtual memory exhausted (in reply to end of DATA command)) Has anybody seen this already? Which memory lmtpd claims to be exhausted? With no process limits at all? Pascal -- Pascal Gienger, Kanzleistrasse 14, D-78462 Konstanz Postfix+SASL-AUTH+LDAP Solaris 8: http://pgienger.de/postfix/
Authentication problems, "Virtual memory exhausted ... line 125."
Hello all,
I am having trouble configuring Cyrus in any way that will allow me to
authenticate. I've tried doing this several different ways according to the
docs, the HOWTO, the O'Reilly IMAP book, and various list posts. I keep
getting incorrectly rejected when I try to login using imtest and the default
authorization. When I try to login using cyradm, I get a Perl error "Virtual
memory exhausted...line 125". This is totally confusing.
Rather than explain all the details, I have created a clean installation as
an example. I'm sorry this is so long. The kill-cyrus.py script at the
beginning just deletes almost every file or directory on the disk containing
either "cyrus" or "sasl". This is as good as I can do for an uninstall.
Eventually, we hope to allow connection via TLS and authentication against a
remote LDAP server or via PAM. For now, I'm trying to see that it can work
in the simplest configuration.
My environment...
Linux, x86, Red Hat 7.1 with recent updates
Berkeley DB 3.1.17 (as installed with OS)
Cyrus IMAP 2.0.15
Cyrus SASL 1.5.24
Shadow passwords, no kerberos
A few specific questions...
Have I made some gross mistake here that would be causing these woes?
What is the meaning of the Perl error I'm seeing?
Why is it that I can log in using "login" but not with "authenticate"?
Does SASL take the place of PAM or does it act as an interface to PAM?
Would it help to try the RPM for Cyrus IMAP? ... SASL?
What can I change to make this work? ...the million dollar question :-)
If anyone can provide any insight, I would be very, very grateful. I have
been working at this for well over a week. I am new to mail servers so it is
very possible that I don't understand something basic.
Thank you.
Alex Quinn
###
###
[root@localhost /root]# ./kill-cyrus.py
[root@localhost /root]# cd download/cyrus/cyrus-sasl-1.5.24
[root@localhost cyrus-sasl-1.5.24]# ./configure --with-pwcheck=/var/pwcheck
--prefix=/usr --disable-gssapi --disable-krb4 --quiet
[...]
[root@localhost cyrus-sasl-1.5.24]# make all; make install
[...]
[root@localhost cyrus-sasl-1.5.24]# cd ../cyrus-imapd-2.0.15
[root@localhost cyrus-imapd-2.0.15]# ./configure --with-auth=unix
--with-sasldir=/usr --with-openssl=/usr --with-com_err --disable-krb4
--disable-gssapi --quiet
[...]
[root@localhost cyrus-imapd-2.0.15]# make all CFLAGS=-O; make install
[...]
[root@localhost cyrus-imapd-2.0.15]# cp master/conf/normal.conf
/etc/cyrus.conf
[root@localhost cyrus-imapd-2.0.15]# useradd -g mail cyrus
[root@localhost cyrus-imapd-2.0.15]# passwd cyrus
Changing password for user cyrus
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully
[root@localhost cyrus-imapd-2.0.15]# su cyrus
[cyrus@localhost cyrus-imapd-2.0.15]$ tools/mkimap
reading configure file...
i will configure directory /var/imap.
i saw partition /var/spool/imap.
done
creating /var/imap...
creating /usr/sieve...
creating /var/spool/imap...
done
[cyrus@localhost cyrus-imapd-2.0.15]$ exit
exit
[root@localhost cyrus-imapd-2.0.15]# cat /etc/imapd.conf
configdirectory: /var/imap
partition-default: /var/spool/imap
admins: root cyrus
sasl_pwcheck_method: pwcheck
servername: alex.olympus.net
# allowanonymouslogin: yes
allowplaintext: yes
[root@localhost cyrus-imapd-2.0.15]# chown cyrus /etc/imapd.conf
[root@localhost cyrus-imapd-2.0.15]# ll /etc/imapd.conf
-rw-r--r--1 cyrusroot 186 Aug 24 01:44 /etc/imapd.conf
[root@localhost imap]# pwcheck
[root@localhost imap]# /usr/cyrus/bin/master &
[1] 16106
[root@localhost cyrus-imapd-2.0.15]# su cyrus
[cyrus@localhost cyrus-imapd-2.0.15]$ imtest localhost
C: C01 CAPABILITY
S: * OK alex.olympus.net Cyrus IMAP4 v2.0.15 server ready
S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID
NO_ATOMIC_RENAME UNSELECT MULTIAPPEND SORT THREAD=ORDEREDSUBJECT
THREAD=REFERENCES IDLE
S: C01 OK Completed
Password:
C: L01 LOGIN cyrus {8}
+ go ahead
C:
L01 OK User logged in
Authenticated.
Security strength factor: 0
. logout
* BYE LOGOUT received
. OK Completed
Connection closed.
[cyrus@localhost cyrus-imapd-2.0.15]$ ll /etc/sasldb
ls: /etc/sasldb: No such file or directory
[cyrus@localhost cyrus-imapd-2.0.15]$ cyradm localhost
Please enter your password:
IMAP Password: localhost.localdomain>
localhost.localdomain> exit
[cyrus@localhost cyrus-imapd-2.0.15]$ ll /etc/sasldb
ls: /etc/sasldb: No such file or directory
[cyrus@localhost cyrus-imapd-2.0.15]$ /usr/sbin/saslpasswd -c cyrus
Password:
Again (for verification):
saslpasswd: generic failure
[cyrus@localhost cyrus-imapd-2.0.15]$ ll /etc/sasldb
ls: /etc/sasldb: No such file or directory
[cyrus@localhost cyrus-imapd-2.0.15]$ exit
exit
[root@localhost cyrus-imapd-2.0.15
