Re: discrepancies between observed disk usage and reported disk space
On Tuesday 13 June 2006 14:52, Aaron Bennett wrote: > Any thoughts? I think most of this is caused by the metadata maintained by cyrus. I have similar discrepencies on my systems. wt -- Warren Turkal, Research Associate III/Systems Administrator Colorado State University, Dept. of Atmospheric Science Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defeated about Murder.....
On 18 Jun 2006, at 16:45, Andrzej Kwiatkowski wrote: Could you write for what are the others patches ? There are a couple of bug fixes. Several are additional logging, particularly of message delivery and message deletion. These two are serious improvements: http://cvs.itd.umich.edu/cgi-bin/cvsweb.cgi/lfs/cyrus-imap23/ auth_canonifyid.diff?rev=1.1 This patch preserves the krb5_context between subsequent calls. In an environment where imapd must traverse 100Ks mailboxes in order to generate an individual user's mailbox list, krb5_init_context() causes undue load. This routine is already non-reentrant, so the additional static doesn't matter. http://cvs.itd.umich.edu/cgi-bin/cvsweb.cgi/lfs/cyrus-imap23/ ctl_mboxlist.diff?rev=1.1 Adds the following features to ctl_mboxlist: -R When run on a backend, include the backend servers name in the dump. When run on the mupdate master with -u, set the "remote" flag on the mailboxes. Useful for building master mailboxes.db. -V When run on a backend, verify that the mailboxes actually exist on the disk. Useful for correcting master / backend / filesystem corruption. There are a couple more, but they are more difficult to tease out. :wes Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: defeated about Murder.....
>You're options are to >1. Have lmtpproxyd query the local server for each incoming message. >This decreases load on the murder master. You do this by creating a >config file for lmtpproxyd and setting murder master to the >localhost. Then you just need to get the auth setup. Can you show me how to configure lmtpproxyd to use localhost as murder master? Thanks -Jack Xue Quoting Andrzej Kwiatkowski <[EMAIL PROTECTED]>: > Hi.. > > I'was thinking that Murder is a very good concept for cyrus... > Till today. > > I've started some performance test: > > I've 4 MTA with Postfix+Cyrus frontend 2.2.12 > 4 backend with Cyrus 2.3.6 > and 1 Mupdate with cyrus 2.2.12. > > I've started with smtp-stone sending > 3000 msg (1000 for each of 3 users on 1 backend). > > Backend load was very low. > > But in this time mupdate have load about 1,5 (why ?) > The concept was that mupdate is only for changing location of > mailboxes... My frontend have information on mailboxes (checked > with > ctl_mboxlist -d) but they still are looking in mupdate which > causes high load > and SIGSEGV... > Which causes lmtpproxy to drop connecitons and growing queue in > postfix... > > Am i doing something wrong ? > Authentication is doing by sasl digest-md5 to avoid SQL database > performance, > so i think this is no problem.. > > So why mupdate causes such problems ? > > Thanks > AK > > Cyrus Home Page: http://asg.web.cmu.edu/cyrus > Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu > List Archives/Info: > http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defeated about Murder.....
On 18 Jun 2006, at 16:07, Andrzej Kwiatkowski wrote: could you send Your patch, please ? Sure. Look here: http://cvs.itd.umich.edu/cgi-bin/cvsweb.cgi/lfs/cyrus-imap23/ lmtpproxyd.diff?rev=1.1 These are actually against Cyrus IMAP 2.2.x, tho I'm in the process of moving them forward to 2.3.x. :wes Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defeated about Murder.....
2006/6/18, Wesley Craig <[EMAIL PROTECTED]>: On 18 Jun 2006, at 16:07, Andrzej Kwiatkowski wrote: > could you send Your patch, please ? Sure. Look here: http://cvs.itd.umich.edu/cgi-bin/cvsweb.cgi/lfs/cyrus-imap23/ lmtpproxyd.diff?rev=1.1 These are actually against Cyrus IMAP 2.2.x, tho I'm in the process of moving them forward to 2.3.x. Thanks, on frontends i want to use 2.2. becuase of deletion bug which is unresolved. So this patch is a great for me . It was possible to force postfix to send directly to backend, but this was next query to SQL ... I'll try this patch :-) Thanks Could you write for what are the others patches ? AK Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defeated about Murder.....
2006/6/18, Wesley Craig <[EMAIL PROTECTED]>: We have patches to lmtpproxyd to refer to the local mailboxes.db, which greatly improves performance. :wes Hi, could you send Your patch, please ? Thanks AK Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defeated about Murder.....
We have patches to lmtpproxyd to refer to the local mailboxes.db, which greatly improves performance. :wes On 18 Jun 2006, at 09:18, [EMAIL PROTECTED] wrote: You have lmtpproxyd set up to query the murder master server for each message that comes in. You're options are to 1. Have lmtpproxyd query the local server for each incoming message. This decreases load on the murder master. You do this by creating a config file for lmtpproxyd and setting murder master to the localhost. Then you just need to get the auth setup. 2. By pass the frontends, and have postfix deliver directly to the backends We did 1 for awhile, but would still have occasional (but a lot fewer) problems (this is ~ 40K/hr of messages). We then switched to 2 (but using sendmail). We have the user/backend information in our id system and just create a sendmail alias file for this. This has worked for months without problems. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mailbox name diferent than login name in IMAP/POP, possible?
Quoting Igor Brezac <[EMAIL PROTECTED]>: On Sun, 18 Jun 2006, Pedro Algarvio wrote: Is it possible to login like [EMAIL PROTECTED] and get the mailbox for that user to be for example username_domain.tld? I'm using ldap as the saslauthd backend, so would it be possible to translate [EMAIL PROTECTED] to username_realm, that way users could login with their normal email address to services like pop/imap. I know authentication is possible, that's plain ldap, but the mailbox "translation" is my doubt. P.S.: I'd like to get this without using cyrus virtdomains setting, or at least leaving it to off. You will need to write/hack code for this to work. You have several options: write a custom authorization module (it can be ldap driven) or hack an existing authorization module, hack ptloader/ldap code or write your own ptloader plugin, or write a custom canon sasl plugin. -- Igor Where exacly should I "hack" the code to achieve this? And how exactly can I tell cyrus to use a mailbox name diferent from the username, will a PTLOADER be able to do this? To make things harder, I don't know C, I'm just reading documentation as need, although I might get some help from somewhone who knows but he's not that aquainted to cyrus, so any starting points would be wonderfull. Perhaps another config var could be added to the ldap PTLOADER so that we can provide the maildrop ldap attribute name, besides maildrop of course(this might be another way). Well, what I think the ldap ptloader could gain is give the user the ability to provide the atribute he wants to retain from an ldap query. Just some other ideas to improve your ptloader. Thanks! -- Pedro Algarvio __ ___ ___ ______ __ | Y .' _| _ .-.' _| |_ .-..-. |. | | _| 1___| _ | _| _|__| _ | _| _ | |. | |__| | |_|__| ||__|_|__| |___ | |: 1 ||: 1 | |_| |::.. . ||::.. . | ufs [AT] ufsoft [DOT] org `---'`---'ufs [AT] sapo [DOT] pt Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Move data to new harddisk (fs-type and options, config)
Thomas Schlosser skrev: Thanks Simon, this helped me a lot. I will now create 2 disk partitions (in an extended part I think) one for configdirectory and one for partition-default and mount them to their old loacions. I will use rsync -aH instead of cp btw I forgot to mention that this is a SUSE box (9.2, 2.6 kernel) where Reiser is the default FS. I would use LVM at least for the /var/spool/imap directory. With LVM you may easely increase the size of /var/spool/imap. //Kamijo Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defeated about Murder.....
You have lmtpproxyd set up to query the murder master server for each message that comes in. You're options are to 1. Have lmtpproxyd query the local server for each incoming message. This decreases load on the murder master. You do this by creating a config file for lmtpproxyd and setting murder master to the localhost. Then you just need to get the auth setup. 2. By pass the frontends, and have postfix deliver directly to the backends We did 1 for awhile, but would still have occasional (but a lot fewer) problems (this is ~ 40K/hr of messages). We then switched to 2 (but using sendmail). We have the user/backend information in our id system and just create a sendmail alias file for this. This has worked for months without problems. -Patrick Quoting Andrzej Kwiatkowski <[EMAIL PROTECTED]>: > Hi.. > > I'was thinking that Murder is a very good concept for cyrus... > Till today. > > I've started some performance test: > > I've 4 MTA with Postfix+Cyrus frontend 2.2.12 > 4 backend with Cyrus 2.3.6 > and 1 Mupdate with cyrus 2.2.12. > > I've started with smtp-stone sending > 3000 msg (1000 for each of 3 users on 1 backend). > > Backend load was very low. > > But in this time mupdate have load about 1,5 (why ?) > The concept was that mupdate is only for changing location of > mailboxes... My frontend have information on mailboxes (checked > with > ctl_mboxlist -d) but they still are looking in mupdate which > causes high load > and SIGSEGV... > Which causes lmtpproxy to drop connecitons and growing queue in > postfix... > > Am i doing something wrong ? > Authentication is doing by sasl digest-md5 to avoid SQL database > performance, > so i think this is no problem.. > > So why mupdate causes such problems ? > > Thanks > AK > > Cyrus Home Page: http://asg.web.cmu.edu/cyrus > Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu > List Archives/Info: > http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Move data to new harddisk (fs-type and options, config)
Thanks Simon, this helped me a lot. I will now create 2 disk partitions (in an extended part I think) one for configdirectory and one for partition-default and mount them to their old loacions. I will use rsync -aH instead of cp btw I forgot to mention that this is a SUSE box (9.2, 2.6 kernel) where Reiser is the default FS. Cheers Thomas2006/6/18, Simon Matter <[EMAIL PROTECTED]>: > Hi,> I am about to move my IMAP storage to a new harddisk (old HD gets too> small> and may fail in the near future).> Cyrus serves a small group of people (about 5), running on old hardware > (PII,300) with a new 80GB IDE HD. No performance issues.> I have several questions about details of the migration:>> 1. a) FS-type: any recommendations (I would have used reiserfs, but ext3 > or> ext2 could be options as well)?Okay, this lets us know that you are running Linux. I'm using XFS and ext3and it seems that reiserfs also performs well with cyrus. I think you canchoose any of them, however not every FS is well integrated in any distro. > b) FS options: should I use specials options when generating the FS (there> may be a lot of files in folders as we all know), eg inodes,...I don't think you have to care about that for a small 80G cyrus message store. Of course that's something else in the multi Terabyte range...> c) any mount-options I should use?> FYI: the new harddisk will be mounted to /data and will have directories> imap/part and imap/conf directories for the partition and configdir) I'm usually using noatime for /var/spool/imap with ext3.>> 2. Which parts should I move:> a) Of course I want to move the partition (partition-default, which ist> /var/spool/imap currently) > b) I would like to move configdirectory (/var/lib/imap) as well - is the> recommendable?>> 3. How to do the migration> I intended to> - stop postfix and imapd> - copy all files with "cp --preserve --recursive" to the new location (are > there links to be taken care of)?Yes, there may be hardlinks when using single instance store, "rsync -aH"is better anyway.> - change /etc/imapd.conf regarding the two settings (partition-default and > configdirectory)I don't suggest putting configdirectory and partition-default on the samefilesystem if you want to use noatime option.> - start imapd and postfix again> Will this work or is this not the preferred way to to it This will work but why not simply move the data to filesystems on the newdisk and then mount them to the old locations?Simon Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
defeated about Murder.....
Hi.. I'was thinking that Murder is a very good concept for cyrus... Till today. I've started some performance test: I've 4 MTA with Postfix+Cyrus frontend 2.2.12 4 backend with Cyrus 2.3.6 and 1 Mupdate with cyrus 2.2.12. I've started with smtp-stone sending 3000 msg (1000 for each of 3 users on 1 backend). Backend load was very low. But in this time mupdate have load about 1,5 (why ?) The concept was that mupdate is only for changing location of mailboxes... My frontend have information on mailboxes (checked with ctl_mboxlist -d) but they still are looking in mupdate which causes high load and SIGSEGV... Which causes lmtpproxy to drop connecitons and growing queue in postfix... Am i doing something wrong ? Authentication is doing by sasl digest-md5 to avoid SQL database performance, so i think this is no problem.. So why mupdate causes such problems ? Thanks AK Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Move data to new harddisk (fs-type and options, config)
> Hi, > I am about to move my IMAP storage to a new harddisk (old HD gets too > small > and may fail in the near future). > Cyrus serves a small group of people (about 5), running on old hardware > (PII,300) with a new 80GB IDE HD. No performance issues. > I have several questions about details of the migration: > > 1. a) FS-type: any recommendations (I would have used reiserfs, but ext3 > or > ext2 could be options as well)? Okay, this lets us know that you are running Linux. I'm using XFS and ext3 and it seems that reiserfs also performs well with cyrus. I think you can choose any of them, however not every FS is well integrated in any distro. > b) FS options: should I use specials options when generating the FS (there > may be a lot of files in folders as we all know), eg inodes,... I don't think you have to care about that for a small 80G cyrus message store. Of course that's something else in the multi Terabyte range... > c) any mount-options I should use? > FYI: the new harddisk will be mounted to /data and will have directories > imap/part and imap/conf directories for the partition and configdir) I'm usually using noatime for /var/spool/imap with ext3. > > 2. Which parts should I move: > a) Of course I want to move the partition (partition-default, which ist > /var/spool/imap currently) > b) I would like to move configdirectory (/var/lib/imap) as well - is the > recommendable? > > 3. How to do the migration > I intended to > - stop postfix and imapd > - copy all files with "cp --preserve --recursive" to the new location (are > there links to be taken care of)? Yes, there may be hardlinks when using single instance store, "rsync -aH" is better anyway. > - change /etc/imapd.conf regarding the two settings (partition-default and > configdirectory) I don't suggest putting configdirectory and partition-default on the same filesystem if you want to use noatime option. > - start imapd and postfix again > Will this work or is this not the preferred way to to it This will work but why not simply move the data to filesystems on the new disk and then mount them to the old locations? Simon Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Move data to new harddisk (fs-type and options, config)
Hi, I am about to move my IMAP storage to a new harddisk (old HD gets too small and may fail in the near future). Cyrus serves a small group of people (about 5), running on old hardware (PII,300) with a new 80GB IDE HD. No performance issues. I have several questions about details of the migration: 1. a) FS-type: any recommendations (I would have used reiserfs, but ext3 or ext2 could be options as well)? b) FS options: should I use specials options when generating the FS (there may be a lot of files in folders as we all know), eg inodes,... c) any mount-options I should use? FYI: the new harddisk will be mounted to /data and will have directories imap/part and imap/conf directories for the partition and configdir) 2. Which parts should I move: a) Of course I want to move the partition (partition-default, which ist /var/spool/imap currently) b) I would like to move configdirectory (/var/lib/imap) as well - is the recommendable? 3. How to do the migration I intended to - stop postfix and imapd - copy all files with "cp --preserve --recursive" to the new location (are there links to be taken care of)? - change /etc/imapd.conf regarding the two settings (partition-default and configdirectory) - start imapd and postfix again Will this work or is this not the preferred way to to it Thanks for any hints! Cheers Thomas Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mailbox name diferent than login name in IMAP/POP, possible?
Quoting Rudy Gevaert <[EMAIL PROTECTED]>: Igor Brezac wrote: On Sun, 18 Jun 2006, Pedro Algarvio wrote: Is it possible to login like [EMAIL PROTECTED] and get the mailbox for that user to be for example username_domain.tld? I'm using ldap as the saslauthd backend, so would it be possible to translate [EMAIL PROTECTED] to username_realm, that way users could login with their normal email address to services like pop/imap. I know authentication is possible, that's plain ldap, but the mailbox "translation" is my doubt. P.S.: I'd like to get this without using cyrus virtdomains setting, or at least leaving it to off. You will need to write/hack code for this to work. You have several options: write a custom authorization module (it can be ldap driven) or hack an existing authorization module, hack ptloader/ldap code or write your own ptloader plugin, or write a custom canon sasl plugin. You could put an imap/pop3 proxy before it. e.g perdition. Rudy I've came up with this issue to avoid perdition the most I can. Thanks anyway. -- Pedro Algarvio __ ___ ___ ______ __ | Y .' _| _ .-.' _| |_ .-..-. |. | | _| 1___| _ | _| _|__| _ | _| _ | |. | |__| | |_|__| ||__|_|__| |___ | |: 1 ||: 1 | |_| |::.. . ||::.. . | ufs [AT] ufsoft [DOT] org `---'`---'ufs [AT] sapo [DOT] pt Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Mailbox name diferent than login name in IMAP/POP, possible?
Igor Brezac wrote: On Sun, 18 Jun 2006, Pedro Algarvio wrote: Is it possible to login like [EMAIL PROTECTED] and get the mailbox for that user to be for example username_domain.tld? I'm using ldap as the saslauthd backend, so would it be possible to translate [EMAIL PROTECTED] to username_realm, that way users could login with their normal email address to services like pop/imap. I know authentication is possible, that's plain ldap, but the mailbox "translation" is my doubt. P.S.: I'd like to get this without using cyrus virtdomains setting, or at least leaving it to off. You will need to write/hack code for this to work. You have several options: write a custom authorization module (it can be ldap driven) or hack an existing authorization module, hack ptloader/ldap code or write your own ptloader plugin, or write a custom canon sasl plugin. You could put an imap/pop3 proxy before it. e.g perdition. Rudy Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
