Re: DB4 error?
Le mer 21/04/2004 à 21:38, Noll Janos a écrit : > > Apr 21 11:07:19 lmtpunix[16815]: DBERROR db4: Logging region out of > memory; you may need to increase its size I found something in the DB4 documentation: int DB_ENV->set_lg_regionmax(DB_ENV *dbenv, u_int32_t lg_regionmax); Description Set the size of the underlying logging subsystem region, in bytes. By default, or if the value is set to 0, the base region size is 60KB. The log region is used to store filenames, and so may need to be increased in size if a large number of files will be opened and registered with the specified Berkeley DB environment's log manager. [...] I think I'll try this... -- | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! | --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: DB4 error?
Hi! Le ven 23/04/2004 à 12:32, Craig Ringer a écrit : > > > > Apr 21 11:07:19 lmtpunix[16815]: DBERROR db4: Logging region out of > > memory; you may need to increase its size > > -- > > If you can, I'd stop the master and then run db_verify on deliver.db. I > wouldn't be surprised to see it fail. Thank you, I'll try that next time. > When I encountered deliver.db corruption, I simply stopped the master, > verified the deliver db (it was corrupt), moved deliver.db and the > contents of the db/ directory out of the way, and started the master > again. Yes, I do know the "quick solution", but the problem keeps coming back every few days... > I'd also take this opportunity to save a plaintext copy of the mailboxes > list with 'ctl_mboxlist -d > /var/imap/mailboxes_`date -I`.txt' (or > similar). I'm not using Berkeley DB for mailboxes, but that's another story :-) So I don't fear that. With Cyrus 2.1.16 and DB3 (3.2?) I didn't have this problem. I do use a "custom patch" that lets me place the delivery database in another directory (that's symlinked to another partition), but I don't think that's a problem... -- | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! | --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
DB4 error?
Hi! Doest anyone have any ideas for the following problem? Apr 21 11:07:19 lmtpunix[16815]: DBERROR db4: Logging region out of memory; you may need to increase its size Apr 21 11:07:19 lmtpunix[16815]: DBERROR: opening /var/imap/deliver.db: Cannot allocate memory Apr 21 11:07:19 lmtpunix[16815]: DBERROR: opening /var/imap/deliver.db: cyrusdb error Apr 21 11:07:19 lmtpunix[16815]: FATAL: lmtpd: unable to init duplicate delivery database -- The system works fine, up to a point, then it goes haywire... Debian Sarge Cyrus 2.2.3 Berkeley DB 4.2.52-16 Thank you! -- | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! | -- | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! | --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: interesting limitation
Hi! Yes, and ext2/3 has the same limitation. Ext2/3 code in the Linux kernel can be modified (one line) and recompiled to allow more directories, up to cca. 65000. ReiserFS is known not to have this limitation, if I'm right. But the solution should be a (configurable) more scalable hashing method, for a large user base. On Sat, 2003-03-29 at 20:04, Jure Pecar wrote: > Hi all, > > Recently i was testing a 2.2 branch on linux with Veritas vxfs. I wanted to > create 20 users in the form of userN, where n is 1..20. I soon found > out that vxfs won't let me create more than 32k subdirs in one dir. > > This is clearly a limitation of the filesystem. How does other filesystems > handle this? > > The solution here is full dir hash. But, the next limit is at 26*32k users. > Is anyone actually nearing this number of users on a single box? Probably > not, but who knows what the future may bring ... -- | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! |
RE: PostgreSQL backend: a waste of time?
Hy! I think that's a very good idea, but we found that MySQL is much faster than Postgres, when there are no complex queries (this is the case here), so it might be a better idea to use MySQL. On 25-Nov-2002 Nicola Ranaldo wrote: > Due to our historical problems using BerkeleyDB4 over True64Unix I'm > coding a PostgreSQL backend. > It's in alpha stage and seems to work fine over an our production > server > with about 7500 mailboxes. > Cyrus.log is DBERROR free, and users do not report any problem. > Access to mboxlist is about 5 time faster then with BerkeleyDB4. > If the cyrus community is interested in this little piece of code (i > don't > know due to presence of skiplist backend) we can start a thread now! | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! |
RE: mbdaemon
Hy! I was the one who wrote that :-) It isn't really for load distributing, and it works on only one machine. The goal was to keep all mailbox-folder-data in memory (the mailboxes.db), thus having a very very fast access to it. The old (cyrus 1.5.x) plaintext file mailboxes.db was slow, and even, the newer DB3 backends had problems, with either lockin/waiting for a lock, or by reading/writing the db files, which is always slower, than just seeking in the memory. In theory, ACAP should be the solution for your problems. With an ACAP server, all metadata (folder names, rights, etc.) are kept separately, and are available to all IMAP servers. Maybe you'll have to combine that with Murder to distribute the load. Anyway, ACAP was not available back when I wrote that daemon. I haven't yet used ACAP, but it should be promising. On 14-Mar-2002 Mathieu Arnold wrote: > What do you think of : > http://opensource.prim.hu/mbdaemon/ > I don't really understand it's goals. > I found it looking for a way to distribute my load across many servers. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! |
Re: Erroneous "No space left on device" error
Hi!
If I interpret it right:
1. a /var/spool/imap/user/franki/cyrus.header.NEW file is created
2. attempt is being made to write data to this file
3. attempt fails (operating system returns no space left on device),
so this doesn't seem to be a Cyrus error.
Are the two mailboxes (good and bad) on the same partition?
What happens, if you "su cyrus", and try to run an
"echo blabla > /var/spool/imap/user/franki/cyrus.header.NEW" command?
(I think it should fail also.)
Maybe there's a filesystem corruption? (Though unlikely.)
On 21-Jan-2002 Fred Bacon wrote:
>
> fstat(5, {st_mode=S_IFREG|0600, st_size=135, ...}) = 0
> stat("/var/spool/imap/user/franki/cyrus.header", {st_mode=S_IFREG|0600,
> st_size=135, ...}) = 0
> chdir("/var/spool/imap/user/franki")= 0
> stat("/var/imap/quota/f/user.franki", 0xbfffc3d8) = -1 ENOENT (No such
> file
> or directory)
> stat("/var/imap/quota/u/user", 0xbfffc3d8) = -1 ENOENT (No such file or
> directory)
> open("/var/spool/imap/user/franki/cyrus.header.NEW",
> O_RDWR|O_CREAT|O_TRUNC, 0666) = 6
> fstat(6, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
> old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1,
> 0) = 0x4019
> write(6, "\241\2\213\rCyrus mailbox header\n\"The be"..., 144) = -1
> ENOSPC
> (No space left on device)
>
>
> The error seems to take place when writing to the cyrus.header file.
>
> On the other hand. If I run this for my own mailbox, there this no
> error!
> The last line is
>
> write(6, "\241\2\213\rCyrus mailbox header\n\"The be"..., 143) = 143
>
> There is no obvious difference between the two mailboxes. Does this
> suggest anything to anyone?
| Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu |
| "Expect the unexpected!"| ICQ# 4547866 | Be free! |
Re: Erroneous "No space left on device" error
Hy! On 20-Jan-2002 Fred Bacon wrote: > left on device" error message. Does this make sense? Since > reconstruct gave the same error message, we may yet have another > problem. I wouldn't think that reconstruct would care about ACLs. :-\ This is good! That means you can easily reproduce the error message with reconstruct. Now do a system call trace on reconstruct (strace -o logfile.txt reconstruct blabla... on Linux), and maybe this can help you narrow down the problem. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Be free! |
Re: Trying to move mailboxes to different partition
Hy! On 11-Jan-2002 Chris Peck wrote: > You could change the entry in the imapd.conf to the new location. I don't think you can do this for just one letter! Yes, you can create "partitions" in the config, but then you have to move each folder to the new partition from within cyrus, but that's much more slower and much harder. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu |
RE: Trying to move mailboxes to different partition
Hy! On 11-Jan-2002 Richmond Dyes wrote: > I have Cyrus 2.0.9 running on RH 7.1. I want to move some of my mail > users to another partition because it is getting full. I downloaded the [...] > 2. Is there a way to maunally move a user? If you use a hashed IMAP spool (you probably do), moving by hand is easy. A way for it: 1. stop the cyrus imap server processes 2. move a letter and subdirectories, like "/var/spool/imap/a" to "/mnt/otherhd/somethingimap/a" (make sure the access rights are kept proper!) 3. make a symlink "/var/spool/imap/a" to "/mnt/otherhd/somethingimap/a" 4. start the imap server You can move by first letters this way, which is pretty easy to handle, and efficient. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: Reliable mailstore
Hello! On 10-Jan-2002 [EMAIL PROTECTED] wrote: > Upon inexpensivety: Maybe there is another idea possible. Kimberlite > does sort of a parallel mount to a RAID on a shared SCSI bus. Maybe > there is a way here to achieve low-level distance. If you have a bus > that could be transported over some length - say: fiberchannel over ATM? - > there would be no need to bother what the OS does on it. And also, there is a theoretical "third" way: The IMAP aggregator/proxy should connect to not one, but two boxes (the "appropriate" one for the user, and the backup of that one). Whenever there's a "modifying" command (like "DELETE"), it should execute that on both the live box and the hot backup. If a "FETCH" command arrives, that can be executed on any one of the boxes. This way, the mail stores should be synchronized. But you must also ensure that when one box disconnects, dies, and then reconnects, it has to synchronize. And, of course, incoming smtp/lmtpd mail must go to both boxes. This works somewhat like RAID1 with hard disks. Read-only operations should be normal speed (though FETCH BODY is not 100% read only, it sets the Seen flag), write could be paralellized, thus also be at about normal speed. There are catches, though. If a mail arrives at the live box, then the user deletes the "last" mail in the folder, and only then (in time) does the same mail arrive in the backup box, you have two desynched boxes. This shouldn't really happen, since the two boxes should run at the same speed, and the live box has somewhat larger load, but cases like this should be considered. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: Sieve again on cyrus 2.0.16
Hy! On 07-Jan-2002 Ramiro Morales wrote: > } will be stored (I'm putting them in /var/imap/sieve; the default > } Cyrus uses is /usr/cyrus). > > Wrong, the default is /usr/sieve By the way, I think the default "/usr/sieve" path for sieve filter files is a wrong and illogical choice. It should go under "/var/sieve" or "/var/imap/sieve" instead, by default, because of the nature of the data contained. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: FWD: cyrus 2.0.16/SIEVE
Hi!
On 07-Jan-2002 Abu @ Trabas Dot Com wrote:
> i want apache run installsieve/sieveshell with exec command, example:
> exec('installsieve -u user@domain -i file -a file server');
>
> -> how to supply password if apache has no password or know password.
> the password fill from prompt shell?
I think sieveshell would work, if you opened a pipe to it, and wrote the
password, then the commands to it.
The other way (example):
1. create a temp file, like /tmp/example1 that contains
thepasswordhere
list
2. use the command:
cat /tmp/example1 | sieveshell -u usernamehere 127.0.0.1
And voila!
The first approach might be more secure.
| Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu |
| "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
FWD: cyrus 2.0.16/SIEVE bug
Hy!
(Hope you're not seeing this mail twice.)
I might have hit a bug.
I tried a SIEVE script on a test mail that had a
subject: "[Prim]xx" (without quotes, of course)
The script's condition section:
if header :matches "Subject" "[Prim]*"
By common sense, this should match. But it doesn't.
What does match:
if header :matches "Subject" "\\[Prim]*"
And the cause?
Cyrus uses the unix fnmatch() function, which was made for matching filenames.
And that is the reason it has one "side-effect": it interprets not only * and ?
characters, but also [ ] chars.
And from "our" point of view, that's bad.
I've read the RFC, and it doesn't specify that you have to escape the [ char
(and this is only logical).
I've attached a fix, it should be good (also included below).
There may be problems with the "*" characters escaped to "\\*" (that should
instead be "\*", which is in a C source file "\\*"). The RFC is not really clear
about this!
*** comparator.c.original Fri Jan 4 00:49:34 2002
--- comparator.c Fri Jan 4 02:04:59 2002
***
*** 38,43
--- 38,70
#include "tree.h"
#include "sieve.h"
+ /* string_match added by Noll Janos <[EMAIL PROTECTED]> */
+ static int string_match(const char *pat, const char *text)
+ {
+ int ret,nt;
+ char *epat; /* will be the pattern, [-escaped */
+ char *p,*q;
+
+ if (!strchr(pat,'[')) { /* no [ - no problem */
+ ret = !fnmatch(pat, text, 0);
+ } else {
+ /* count how many ['s there are */
+ for ( nt=-1, p=(char *)pat-1 ; p!=NULL ; p=strchr(p+1,'[') ) nt++;
+
+ /* copy and escape ['s */
+ epat=(char *)malloc(strlen(pat)+nt+1);
+ for ( p=(char *)pat,q=epat ; *p ; p++ ) {
+ if (*p=='[') { *(q++)='\\'; }
+ *(q++)=*p;
+ }
+ *q=0;
+
+ ret = !fnmatch(epat, text, 0);
+ free(epat);
+ }
+ return(ret);
+ }
+
/* --- i;octet comparators --- */
/* just compare the two; these should be NULL terminated */
***
*** 93,99
static int octet_matches(const char *pat, const char *text)
{
! return !fnmatch(pat, text, 0);
}
#ifdef ENABLE_REGEX
--- 120,126
static int octet_matches(const char *pat, const char *text)
{
! return string_match(pat, text);
}
#ifdef ENABLE_REGEX
***
*** 146,152
for (i = 0; t[i] != '\0'; i++)
t[i] = toupper(t[i]);
! ret = !fnmatch(p, t, 0);
free(p); free(t);
return ret;
--- 173,179 ----
for (i = 0; t[i] != '\0'; i++)
t[i] = toupper(t[i]);
! ret = string_match(p, t);
free(p); free(t);
return ret;
-
| Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu |
| "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
sievebug.patch
Description: sievebug.patch
RE: Help! How can I change the location of db's from /var/imap/d
Hy! As someone said on this list, you can: 1. stop the cyrus master daemon (stop cyrus alltogether) 2. delete the __db.*** files 3. start the cyrus daemon again But do verify this by scanning through the archive, before doing so :) BTW, I think you can safely symlink the /db directories, provided that cyrus is stopped when doing so. On 16-Dec-2001 Taro Ikai wrote: > My disk partition assigned to /var is not very big. > One of my __db.*** files filled the disk partition, and > stopped imapd from accepting any more e-mails. > > Can somebody tell me how I can change the location > of these __db.*** files? > > Thanks in advance. > > Taro | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: Question: how to disable duplicate delivery mode?
Hy! On 14-Nov-2001 Ian Castle wrote: > Duplicate elimnation is not a runtime or build option. I think it should be. (And, as it seems, I'm not the only one :-) > However, it is probably worth checking that dupelim is used > consistently... It seems to be. But watch out for SIEVE! If dupe elimination is off, so is SIEVE, as the comment says in the lmtpd.c: sieve_find_script(const char *user) function /* duplicate delivery suppression is needed for sieve */ > If it does work, then the thing to do would be to add a config to > imapd.conf to control wether or not you want duplicate elimination. I'm also for that. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: DBERROR db3
Hi! I think, I also experienced similar problems with that release. I think you should try upgrading to the latest version (2.0.16) On 06-Nov-2001 Alex Werner wrote: > > Hallo *, > we are using sol8 with cyrus v2.0.12. I become this strange messages in > > the log: > 26 10:10:43 merkur.ecrc.de imapd[13792]: [ID 866726 local6.error] > DBERROR db3: 104 lockers > Oct 26 10:10:48 merkur.ecrc.de imapd[13814]: [ID 866726 local6.error] > DBERROR db3: 105 lockers > Oct 26 10:10:54 merkur.ecrc.de imapd[13858]: [ID 866726 local6.error] > DBERROR db3: 106 lockers > Oct 26 10:11:05 merkur.ecrc.de imapd[13895]: [ID 866726 local6.error] > DBERROR db3: 107 lockers > Oct 26 10:11:59 merkur.ecrc.de imapd[13988]: [ID 866726 local6.error] | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: Duplicate Delivery Database
Hy! On 04-Sep-2001 Scott Adkins wrote: >> You definitely need to run recovery on your duplicate delivery >> database. I might just delete the database entirely; since the >> duplicate delivery database isn't transactional, it can't guarantee >> consistency, and there's no critical database in it. > > We stopped the server and nuked the delivery files altogether. This only > worked for 2 days, however, and now we are back to where we were. We see > a constant stream of duplicate delivery database errors... I know the following isn't a really nice solution, but if you need to "stabilize" your system fast, you could disable the code that does duplicate delivery checking. I remember it's in one C file, and it's fairly easy to do (just delete or comment out a few dozen lines). Maybe there's even a config option for this now? I repeat, this isn't good for two reasons: - the bug won't be tracked down this way (but bug-hunting shouldn't really be done on production systems anyway) - the duplicate delivery-check feature will be absent of the server (I think you can live with this.) So is this elegant? No! But it _will_ work... | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: Sendmail -> Procmail -> Deliver -> Cyrus
Hy! We're using it this way: --- :0 |/usr/cyrus/bin/deliver -e -a usernamehere -m foldernamehere usernamehere --- On 31-Jul-2001 Mobeen Azhar wrote: > Is there anyone out there using procmail with cyrus imapd 2.x? I have > not been able to use procmail since I moved to the 2.x family because > procmail does not do lmtp delivery and I could not get cyrus' deliver to > work. Right now I am running cyrus imapd 2.0.16 and am curious if the > deliver shipped with it can be invoked from within procmail to deliver > mail to Cyrus. > > Thanks in advance for any information, > --Moby | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: Restoring/Re-Inexing Mailboxes
Hy! Use the "reconstruct" utility, provided with cyrus. It should do the thing, and can be run recursively with the -r switch. On 16-Jun-2001 Klaus wrote: > Still, is there a way to re-index a subfolder or the whole mailbox ? I tried > to add user.test.subfolder to the test.seen file and I tried to re-create the > subfolder using cyradm. Latter one brought back the folder (surprise ...) > - but withouth the messages. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: How to get PAM working
Hy! Maybe you need an RPM package named something like pam-dev, libpam-dev, etc. On 29-May-2001 Stuart Clark wrote: > I can get Cyrus-IMAPd-2.0.13 running well using /etc/passwd for > authentication, and now wish to get it running under PAM. I set up the > /etc/imapd.conf file to have the right pwcheck_method, but it does not want > to work and logs to syslog: [...] > checking for pam_start in -lpam... no > checking PAM support... no <--- | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: unable to delete mailbox
Hy! Do a "sam user.gruppo3 root all" before deleting the mailbox. (sam=set ACL) On 28-May-2001 rj45 wrote: > > localhost.localdomain.org> dm user.gruppo3 > deletemailbox: Permission denied > > I am not able to delete users, anyone would give me a hint ? > I Am root > > cyradm -u root localhost | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: Backuping mail boxes
Hy! On 25-May-2001 Jen-Mei Wu wrote: > Is there a freely available script or program that will backup an IMAP > mail account using the IMAP protocol? I'd like a tool that can backup > an account over the Internet so I can transfer a hosted mail account > to a local server. Maybe fetchmail can be of some use to you? http://tuxedo.org/~esr/fetchmail/ | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
RE: Authentication problems
Hy! Are you sure the SASL libs are in place? I think they're placed in /usr/local/lib, but that might not be in /etc/ld.so.conf (in Linux systems). If this is done, then you need to run ldconfig. If that doesn't help maybe you could try strace-ing the imapd process. And a maybe faster test (well, this is an alternative): $ telnet 127.0.0.1 143 * OK isserver Cyrus IMAP4 v1.6.24 server ready . LOGIN usernamehere passwordhere <-- write this line, starting with the . . OK User logged in . LOGOUT <-- and this On 23-May-2001 Andrew Kirilenko wrote: > Hello! > > I've problems with authentication :( > Can somebody help me with this problem? It's really urgent and important! > > Here's log: > cyrus@isserver:/home/iced$ imtest -p imap -m login localhost > C: C01 CAPABILITY > S: * OK isserver Cyrus IMAP4 v1.6.24 server ready > S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS > X-NON-HIERARCHICAL-RENAME NO_ATOMIC_RENAME UNSELECT ID > S: C01 OK Completed > Password: Terminated > cyrus@isserver:/home/iced$ > cyrus@isserver:/home/iced$ imtest -p imap -m login localhost > C: C01 CAPABILITY > S: * OK isserver Cyrus IMAP4 v1.6.24 server ready > S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS > X-NON-HIERARCHICAL-RENAME NO_ATOMIC_RENAME UNSELECT ID > S: C01 OK Completed > Password: > + go ahead > L01 NO Login failed: generic failure > Authenticated. > Security strength factor: 0 > . logout > * BYE LOGOUT received > . OK Completed > Connection closed. > cyrus@isserver:/home/iced$ | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: FW: Re: mailbox-daemon
Hy! On 21-May-2001 Amos Gouaux wrote: > nj> I think the file close should be equal to a sync. Am I right? > > not necessarily. Ok, I'll fix it. > nj> with 300'000 folders, and 4-5000 active users (daily). Cross your fingers > ;) > > perhaps your users are the ones that should cross their fingers. So far, so good ;) > personally, i don't know why berkeley db still couldn't be used for > robustness of data. just having this daemon cache this data in > memory should in itself be a significant performance boost. To think of it, maybe Berkeley DB could be used. It might even be better... if you can specify it the amount of cache memory to use. But hey, this is also what mb-daemon is about! The daemon side is like a plug-in, you can have a "text file daemon", a Berkeley DB daemon or an SQL-daemon, with modifications and compilation only needed on the daemon side, not on the IMAPd side. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: FW: Re: mailbox-daemon
Hy! You might be right, but fsync() might not be needed. The program deals with the files as follows: - At the start, mailbox.base and mailbox.ext are opened for reading, then the program reads all the data, and closes the files. - 99.9% of the time, the program serves everything from memory. When a new folder is created, an existing is modified, etc, the program opens mailbox.ext for writing (append), appends a line to it, and closes the file. I think the file close should be equal to a sync. Am I right? And so, the "worst" that could happen (with a journalling fs), is that you lose the (new) last line of the file, that's one folder. By the way, since yesterday, the daemon is in live testing now on my system with 300'000 folders, and 4-5000 active users (daily). Cross your fingers ;) Version 0.2 is out, on the page http://opensource.prim.hu/mbdaemon/ (It currently handles only user. folders.) On 20-May-2001 Lawrence Greenfield wrote: > I think this is really interesting work. I haven't looked at the > implementation but have considered this basic idea several times; I > think it's definitely worth the work. > > A random comment on your message: > > The mailbox.ext file is always "small", so writing to it should be >fast, and you could even so sync()-s on it. (But rather, use a >journalling filesystem.) > > All filesystems _require_ you to use fsync(). Journalling filesystems > aren't magical: if you don't fsync() you have no guarantee that the > data gets to disk. So please call fsync; otherwise, you're making > potentially serious problems for people who use your code. > > Larry > >Date: Sun, 20 May 2001 18:17:48 +0200 (CEST) >From: Noll Janos <[EMAIL PROTECTED]> > > However, mailbox.base is "fixed", it doesn't change at (daemon) runtime, >however it could be updated daily or weeky. The mailbox.ext is something > like a >log file, it also contains actions, keys, values, on separate lines. The > daemon >writes changes to (the end) of this file at runtime, whenever a folder is >created, deleted, or changed. The daemon currently handles only "user.*" >folders, this was to save space, and this suits my needs. > > > Any way, if the daemon crashes, no problem, as long as the mailbox.base > and >mailbox.ext files are ok. The mailbox.base is even safer, as it's > read-only for >the daemon. > > About the "write-caching" question: if you use cyrus IMAPd with a text >mailboxes.db, inserting a line into it is always slow, because a new file > is >created (mailboxes.db.NEW), the new line is inserted into it, and then > it's >moved the "old" mailboxes.db. If you use DB3, that could be better, but > even >then, every imapd process must read and process some of the data. > > > But back to my daemon, I must emphasise, that this is only an actual >implementation of the "socket" mail-folder db protocol. The architecture > is >there: one could write a daemon, that uses SQL, a daemon, that does more >caching. > > The picture: > > +---+ ++ > | IMAPD + socket_db |--socket--| mailbox-daemon | > +---+ ++ > > I defined a small "protocol" for the communication. > An example: > IMAPD wants to get the data for the folder "user.test12" > The steps > 1. IMAPd sends a message through the socket: "GET user.test12" > 2. mailbox-daemon replies: "OK user.test12 0 default user.test12 > lps" > > There are a few more functions (iterated GET, SET, REMOVE) that a daemon > must >implement, but that's all. Caching is easy to do, as every GET, SET, etc. > gets >to this daemon AND ONLY this daemon. > > I'll do some real testing over the weekend, and maybe I'll whip up some > more >documentation. >
FW: Re: mailbox-daemon
Oops, I wanted to reply to the list, but instead, I replied to the one posing
the question.
By the way, I found two bugs in my mailbox-daemon, so expect the next version
posted soon (if anyone's interested ;-)
---
Hy!
My mailer-daemon is currently a beta version, or a first version, so it may
even contain conceptual bugs, but much can change.
My concept was the following:
The daemon uses two files, mailbox.base and mailbox.ext. Both contain
key-value pairs (and actions), like the DB3 or text files cyrus currently uses.
However, mailbox.base is "fixed", it doesn't change at (daemon) runtime,
however it could be updated daily or weeky. The mailbox.ext is something like a
log file, it also contains actions, keys, values, on separate lines. The daemon
writes changes to (the end) of this file at runtime, whenever a folder is
created, deleted, or changed. The daemon currently handles only "user.*"
folders, this was to save space, and this suits my needs.
The mailbox.ext file is always "small", so writing to it should be fast, and
you could even so sync()-s on it. (But rather, use a journalling filesystem.)
Any way, if the daemon crashes, no problem, as long as the mailbox.base and
mailbox.ext files are ok. The mailbox.base is even safer, as it's read-only for
the daemon.
About the "write-caching" question: if you use cyrus IMAPd with a text
mailboxes.db, inserting a line into it is always slow, because a new file is
created (mailboxes.db.NEW), the new line is inserted into it, and then it's
moved the "old" mailboxes.db. If you use DB3, that could be better, but even
then, every imapd process must read and process some of the data.
But back to my daemon, I must emphasise, that this is only an actual
implementation of the "socket" mail-folder db protocol. The architecture is
there: one could write a daemon, that uses SQL, a daemon, that does more
caching.
The picture:
+---+ ++
| IMAPD + socket_db |--socket--| mailbox-daemon |
+---+ ++
I defined a small "protocol" for the communication.
An example:
IMAPD wants to get the data for the folder "user.test12"
The steps
1. IMAPd sends a message through the socket: "GET user.test12"
2. mailbox-daemon replies: "OK user.test12 0 default user.test12 lps"
There are a few more functions (iterated GET, SET, REMOVE) that a daemon must
implement, but that's all. Caching is easy to do, as every GET, SET, etc. gets
to this daemon AND ONLY this daemon.
I'll do some real testing over the weekend, and maybe I'll whip up some more
documentation.
On 17-May-2001 Michael Fair wrote:
> This sounds really cool!
>
> Can you explain more of the details on how you implemented it?
> What happens on system crash?
> How often does it "sync" to the hard drive?
> What's the difference between this and using a write-caching
> hard drive scheme?
>
> -- Michael --
>
> - Original Message -
> From: "Noll Janos" <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>
> Sent: Monday, May 14, 2001 5:10 PM
> Subject: mailbox-daemon
>
>
>> Hy!
>>
>> Until now, cyrus could handle either a text file folderlist (mailboxes)
> or a
>> DB3 folderlist (mailboxes.db). But now it can user "socket"
> folder-handler,
>> to connect to a daemon. The mailbox-server "daemon" keeps all the
>> mailbox data ("formerly" mailboxes.db) in memory, so it should be faster.
[...]
mailbox-daemon
Hy!
Until now, cyrus could handle either a text file folderlist (mailboxes) or a
DB3 folderlist (mailboxes.db). But now it can user "socket" folder-handler,
to connect to a daemon. The mailbox-server "daemon" keeps all the
mailbox data ("formerly" mailboxes.db) in memory, so it should be faster. I
tested it with a dataset containing 300 000 folders, and it seemed fast,
especially at create folder/delete folder actions.
If anybody feels like he wants to experiment with my "mailbox-patch", here it
is. It's a 0.1 version, so don't expect much, but it basicly works.
You might need some C knowledge, and it's not yet for the average user!
The URL is: http://opensource.prim.hu/mbdaemon/
If the protocol works, this could have much broader implications. For
example, imagine a "plug-in" daemon that connects to a database and does
caching. Although this could be done by modifying cyrus code, in some systems
it would be faster, as the mailbox-daemon would not need to reconnect to the
database for each cyrus instance and it could live on just one db connection.
I'll try to release a 0.2 version in one or two weeks with more documentation,
but meanwhile, you can test this one.
Any comments, replies will be appreciated.
| Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu |
| "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
Re: forking/db3/performance problem
Hi! On 12-May-2001 Amos Gouaux wrote: > nj> unix domain socket. I think it would be even faster than DB3, because DB3 > has > nj> to access and load (some of) the data and log files for each > cyrus-instance. > > ^^^ > well, here's a total stab in the dark: why not multi-thread it? You mean, multithread the daemon? Or multithread cyrus? If you were to multithread cyrus, you'd have to do it with pop3, lmtpd and the other parts too. It would be nice, but I think that's much work. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
forking/db3/performance problem
Hi! We've experienced the same problem with cyrus 2.0.xx as some of you have, it seem the db3 part can sometimes "deadlock". There is a partial solution: make cyrus use text files, instead of db3. You have to modify one or two lines in the source, but sadly I don't have the source here with me right now. Another (performance) problem: if you use a text mailbox file (mailboxes), it'll get bigger and bigger, slower and slower. Ours is about 20 MByte-s large, and has around 200'000 folders (approx.) The problem is, whenever one wants to create a new folder (or delete one), the new "line" has to be inserted into the file, which requires a full copy, and is slow. DB3 is nice, but the reconstruct is way too slow, as I experienced. I'm planning on doing a "mailbox-daemon", which will load all mailbox data into itself, and persist. It would communicate with the cyrus-server with a unix domain socket. I think it would be even faster than DB3, because DB3 has to access and load (some of) the data and log files for each cyrus-instance. | Noll Janos <[EMAIL PROTECTED]> | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
