Re: Migrating from old PPC server to new AMD cyrus [SOLVED]
Adam D wrote: I have been looking at what tools to use to migrate all email and boxes over to our new server. Old server is running cyrus 2.2.13.11 PPC and the new server is running cyrus 2.2.13.11 AMD64. So far I have been trying to use imapsync but it is giving me all or folder only transfers. There are a lot of shared mailboxes but when I used the standard imapsync commands it copies all shared mailboxes into the user directory which we can not have. I managed to use the --exclude parm but when I do it only creates the mailboxes on the destination. Here is a sample what I am doing: imapsync --syncacls --exclude 'shared1|shared2l' --host1 localhost --user1 user1 --passfile1 file1 --host2 host2 --user2 user2 --passfile2 file2 Are there cyrus commands to migrate the files over to another server? Am I going about this the right way? -Adam Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html I found what I have been looking for. I have been used to the seen files located in /var/lib/cyrus/user but when using virtual domains, cyrus creates a directory in . labeled domain with the proper domains user and quota files located within. Thanks for the help. New location: /var/lib/cyrus/domain/letter/domain/user -Adam Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus on iscsi
On Wed, Feb 27, 2008 at 09:20:39PM -0500, Jeffrey T Eaton wrote: Our system consists of 10 backend servers. Each is a Sun v240 running Solaris 8, with a QLogic iSCSI initiator with two gigabit ethernet links to our SAN network. Each system mounts 4x250 GB paritions for mailbox storage, and one 50GB partition for Cyrus databases, logs, and sieve scripts. If you upgrade to Solaris 10, you can omit the QLogic cards by using the native Iscsi initiator instead. It may actually work better. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus on iscsi
Gary Mills wrote: On Wed, Feb 27, 2008 at 09:20:39PM -0500, Jeffrey T Eaton wrote: Our system consists of 10 backend servers. Each is a Sun v240 running Solaris 8, with a QLogic iSCSI initiator with two gigabit ethernet links to our SAN network. Each system mounts 4x250 GB paritions for mailbox storage, and one 50GB partition for Cyrus databases, logs, and sieve scripts. If you upgrade to Solaris 10, you can omit the QLogic cards by using the native Iscsi initiator instead. It may actually work better. We do have some Solaris 10 machines (not mail servers) using the native software iSCSI initiator. There was some concern about performance, particularly when doing heavy IO, such as for backups. I haven't personally looked to compare the relative performance of the two, however. We will probably be upgrading to Solaris 10 for our backend mailservers in the very near future, because Sun doesn't appear to be selling any more v240's, and the v245's won't boot Solaris 8. We may also seriously consider dropping the Sun hardware entirely, and moving toward Linux. The rest of our mail infrastructure already runs on Linux (frontends, mupdate, and smtp/mx layer), so it seems likely that we will at least consider it. -jeaton Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
possible sieve/cyrus redirect issue?
We use Cyrus IMAP4 v2.2.12-Invoca-RPM-2.2.12-8.1.RHEL4 server ready ESMTP Sendmail 8.13.1/8.13.1; Thu, 28 Feb 2008 09:52:17 -0600 and squirrelmail 1.4 Here is the issue... some of my users login to webmail (squirrelmail), send a message, and a copy is placed in their Sent folder (I can tell is there by looking directly at the mailstore). ... by the log files I can tell when the message entered the system, got an id, duplicate check, mark as Sent, etc. but the message never made it to the destination... eventhough there is a Message accepted for delivery log entry. This problem does not seem to follow a patern and not for the same user. Also, part of the same email message, the log entry below alway show for that user... I don't know what that means sendmail[24713]: m1QIIdVB024713: Authentication-Warning: server.jsums.edu: cyrus set sender to [EMAIL PROTECTED] using -f One of the messages were sent to user that is running sieve to redirect; I can see where the sieve script was contacted, but I can tell were the message was redirected/resent to the final destination Feb 26 12:18:39 server lmtpunix[10876]: duplicate_mark: [EMAIL PROTECTED] [EMAIL PROTECTED] 1204049919 0 how does the duplicate_mark work? I understand that some of these issues might not even be cyrus related, but since you guys know so much, might be able to point me in the right direction. Thanks in advanced. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
sieve vacation
Hi, I made some tests with sieve vacation and it works great ! But the record (timestamp) wrote on deliver.db from expiration time seems only accept the minimum of 3 days ! Is this a feature ?? I'm using cyrus-imapd version 2.2.13 on a debian etch. Thanks. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus on iscsi
On Thu, Feb 28, 2008 at 09:48:00AM -0500, Jeffrey Eaton wrote: We will probably be upgrading to Solaris 10 for our backend mailservers in the very near future, because Sun doesn't appear to be selling any more v240's, and the v245's won't boot Solaris 8. We may also seriously consider dropping the Sun hardware entirely, and moving toward Linux. The rest of our mail infrastructure already runs on Linux (frontends, mupdate, and smtp/mx layer), so it seems likely that we will at least consider it. The Sun T2000 servers make wonderful Cyrus IMAP server backends. These are SPARC and run Solaris 10. Anything built for Solaris 8 will run without modification on them. I recommend them. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal [EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Endgame: Cyrus big install at UC Davis
Pascal Gienger wrote: For x86 the patch will have id 127729-07. For SPARC it will have the major number 127728. My esteemed colleague Nick Dugan, ran some tests against this patch and according to the results it gives the desired results. I expect during our next monthly maintenance cycle, we will switch to the idle nodes in our mail-store clusters with 127728 applied, and confirm the result. We will leave the IDR in place on the current active nodes in case we have to switch back. If someone else comes up with quicker confirmation please post. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
What database format are you using for the mailboxes database? What kind
of storage is the metapartition (usually /var/imap) on? What kind of
storage are your mail partitions on?
--On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson
[EMAIL PROTECTED] wrote:
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Jeff Fookson wrote: is unusably slow. Here are the specifics: You are mighty short on the SPECIFICS of your setup. Expect a slew of questions to elicit this information. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Limit the number of lmtpd daemons to around 10 -- that solved the issue
for me.. We let sendmail handle the queuing. It is more than likely a
locking issue..
Michael Bacon wrote:
What database format are you using for the mailboxes database? What kind
of storage is the metapartition (usually /var/imap) on? What kind of
storage are your mail partitions on?
--On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson
[EMAIL PROTECTED] wrote:
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal [EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Jeff,
Delivery database format can cause this type of problem, among
other databases. For any DB that is updated with contention, use
either BerkeleyDB or Skiplist format. We also had a similar issue
when we did not have the expunge process running and pruning the
delivery database and its size kept growing until it slowed down
the entire system.
Cheers,
Ken
On Thu, Feb 28, 2008 at 02:38:37PM -0700, Jeff Fookson wrote:
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Michael Bacon wrote:
What database format are you using for the mailboxes database? What
kind of storage is the metapartition (usually /var/imap) on? What
kind of storage are your mail partitions on?
Databases are all skiplist. Our mail partition and the metapartition are
both on the same filesystem, as we intended that both be part of the
same drbd mirror. That partition is
a linux software RAID 5 (3 SATA disks). On top of the md layer is the
drbd device; on top of that is an lvm2 logical volume; on top of that is
an ext3 filesystem, mounted
as '/var/imap'. The mail is then in /var/imap/mail and the metadata in
/var/imap/config (and we also have /var/imap/certs for the ssl stuff,
and /var/imap/sieve for sieve scripts).
Thanks.
Jeff Fookson
--On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson
[EMAIL PROTECTED] wrote:
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhDPhone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal [EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Jeff,
Just as a rule of thumb, if you've got problems with Cyrus (or any mail
system), 90% of the time they're related to I/O performance.
I've never seen drbd used for Cyrus, but it looks like other folks have
done it. The combination of drbd+lvm2+ext3 might put you somewhere
unpleasant, but I'll have to let the Linux-heads jump in on that one.
Beyond that, I don't see anything obviously wrong, but maybe someone who's
run it more on Linux can chime in.
-Michael
--On Thursday, February 28, 2008 3:36 PM -0700 Jeff Fookson
[EMAIL PROTECTED] wrote:
Michael Bacon wrote:
What database format are you using for the mailboxes database? What
kind of storage is the metapartition (usually /var/imap) on? What
kind of storage are your mail partitions on?
Databases are all skiplist. Our mail partition and the metapartition are
both on the same filesystem, as we intended that both be part of the same
drbd mirror. That partition is
a linux software RAID 5 (3 SATA disks). On top of the md layer is the
drbd device; on top of that is an lvm2 logical volume; on top of that is
an ext3 filesystem, mounted
as '/var/imap'. The mail is then in /var/imap/mail and the metadata in
/var/imap/config (and we also have /var/imap/certs for the ssl stuff, and
/var/imap/sieve for sieve scripts).
Thanks.
Jeff Fookson
--On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson
[EMAIL PROTECTED] wrote:
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhDPhone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
It may be that the software RAID 5 is your problem. Without the
use of NVRAM for a cache, all of the writes need all 3 disks.
That will cause quite a bottle-neck.
Ken
On Thu, Feb 28, 2008 at 03:36:43PM -0700, Jeff Fookson wrote:
Michael Bacon wrote:
What database format are you using for the mailboxes database? What
kind of storage is the metapartition (usually /var/imap) on? What
kind of storage are your mail partitions on?
Databases are all skiplist. Our mail partition and the metapartition are
both on the same filesystem, as we intended that both be part of the
same drbd mirror. That partition is
a linux software RAID 5 (3 SATA disks). On top of the md layer is the
drbd device; on top of that is an lvm2 logical volume; on top of that is
an ext3 filesystem, mounted
as '/var/imap'. The mail is then in /var/imap/mail and the metadata in
/var/imap/config (and we also have /var/imap/certs for the ssl stuff,
and /var/imap/sieve for sieve scripts).
Thanks.
Jeff Fookson
--On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson
[EMAIL PROTECTED] wrote:
Folks-
I am hoping to get some help and guidance as to why our installation of
cyrus-imapd 2.3.9
is unusably slow. Here are the specifics:
The software is running on a 1.6GHz Opteron with 2Gb memory supporting a
user base of about 400
users. The average rate of arriving mail is on the order of 1-2
messages/sec. The active mailstore
is about 200GB. There are typically about 200 'imapd'
processes at a given time and a hugely varying number of 'lmtpds' (from
about 6 to many hundreds during
times of greatest pathology). System load is correspondingly in the 2-15
range, but can spike to 50-70!
Our users complain that the system is extremely sluggish during the day
when the system is most busy.
The most obvious thing we observe is that both the lmtpds and the imapds
are spending HUGE times waiting
on locks. Even when the system load is only 1-2, an 'strace' attached to
an instance of lmtpd or imapd shows
waits of upwards of 1-2 minutes to get a write lock as shown by the
example below (this is from a trace of an 'lmtpd')
[strace -f -p 9817 -T]
9817 fcntl(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=0}) = 0 84.998159
We strongly suspect that these large times waiting on locks is what is
causing the slowness our users are reporting.
We are under the impression that a single instance of cyrus-imapd scales
well up to about 1000 users (with about 1MB active
memory per 'imapd' process), and so we are baffled as to what might be
going on.
A non-standard aspect of our installation which may have something to do
with the problem is that we are
running cyrus on an lvm2 partition that itself is running on top of
drbd. Thinking that the remote writes
to the drbd secondary might be causing delays, we put the primary in
stand-alone mode so that the drbd layer
was not doing any network activity (the drbd link is running at gigabit
speed on its own crossover cable to
the secondary box) and saw no significant change in behavior. Any issues
due to locking and the lvm2 layer
would, of course, still be present even with drbd's activity reduced to
just local writes.
Can anyone suggest what we might do next to debug the problem further?
Needless to say, our users get
extremely unhappy when trivial operations in their mail clients take
over a minute to complete.
Thank you for any thoughts or advice.
Jeff Fookson
--
Jeffrey E. Fookson, PhDPhone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
Jeffrey E. Fookson, PhD Phone: (520) 621 3091
Support Systems Analyst, Principal[EMAIL PROTECTED]
Steward Observatory
University of Arizona
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Okay, I read over this and I felt worth commenting... There's mention of using MD, DRBD, LVM2, etc... it sounds extremely conviluted and way to complex for what you are needing. When you are doing a read or a write, each thing takes it's time before it gets commited to disk. If you are doing DRBD, you may want to change a few settings you're doing raid5 with 3 sata disks using md and drbd... on top of lvm etc. For Example, quote protocol prot-id On the TCP/IP link the specified protocol is used. Valid protocol specifiers are A, B, and C. Protocol A: write IO is reported as completed, if it has reached local disk and local TCP send buffer. Protocol B: write IO is reported as completed, if it has reached local disk and remote buffer cache. Protocol C: write IO is reported as completed, if it has reached both local and remote disk. /quote From personal experience, I have found that people usually use Protocol C... it's great however it can result in slower writes... which depending on your hardware can be very painful. The fact that you have LVM2 sitting in there, as well as MD that means your average write has to go through DRBD (and wrote to both servers) ... as well as LVM2, and MD before it's actually written... (in a very vague sense) Additionally you can use LVM2 Striping I really won't get into that but that may be more beneficial then a RAID-5 with 3 Disks. There's lots of hints if you read over the archives for speed, I can just tell you from what I have read there is nothing you can do with your complex setup to make it better. My best hint for you would be hardware raid, for one that's a big step, if you really want raid-5, it may be more beneficial to use 4 SATA disks... You can and will expect no matter how good your hardware is (read slow writes) with RAID-5 and MD. I had a Zimbra mailserver with RAID-5 and the best write I could get was 75Mbit, and that was using 8 15k RPM SCSI disks... :( Hardware Raid, remove LVM unless you really need it... remove DRBD unless you totally need it there is other ways to create redundancy that are better then DRBD... It's not that I hate DRBD... I just hate seeing it implemented in places where it just does not belong I don't know if this will make sense, if it doesn't let me know and I'll break it down further if you need it. Lastly, if you could show us some of your syslog to see if there is actually any warnings about '440 lockers in use' or such? Scott On Feb 28, 2008, at 2:54 PM, Michael Bacon wrote: Jeff, Just as a rule of thumb, if you've got problems with Cyrus (or any mail system), 90% of the time they're related to I/O performance. I've never seen drbd used for Cyrus, but it looks like other folks have done it. The combination of drbd+lvm2+ext3 might put you somewhere unpleasant, but I'll have to let the Linux-heads jump in on that one. Beyond that, I don't see anything obviously wrong, but maybe someone who's run it more on Linux can chime in. -Michael --On Thursday, February 28, 2008 3:36 PM -0700 Jeff Fookson [EMAIL PROTECTED] wrote: Michael Bacon wrote: What database format are you using for the mailboxes database? What kind of storage is the metapartition (usually /var/imap) on? What kind of storage are your mail partitions on? Databases are all skiplist. Our mail partition and the metapartition are both on the same filesystem, as we intended that both be part of the same drbd mirror. That partition is a linux software RAID 5 (3 SATA disks). On top of the md layer is the drbd device; on top of that is an lvm2 logical volume; on top of that is an ext3 filesystem, mounted as '/var/imap'. The mail is then in /var/imap/mail and the metadata in /var/imap/config (and we also have /var/imap/certs for the ssl stuff, and /var/imap/sieve for sieve scripts). Thanks. Jeff Fookson --On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson [EMAIL PROTECTED] wrote: Folks- I am hoping to get some help and guidance as to why our installation of cyrus-imapd 2.3.9 is unusably slow. Here are the specifics: The software is running on a 1.6GHz Opteron with 2Gb memory supporting a user base of about 400 users. The average rate of arriving mail is on the order of 1-2 messages/sec. The active mailstore is about 200GB. There are typically about 200 'imapd' processes at a given time and a hugely varying number of 'lmtpds' (from about 6 to many hundreds during times of greatest pathology). System load is correspondingly in the 2-15 range, but can spike to 50-70! Our users complain that the system is extremely sluggish during the day when the system is most busy. The most obvious thing we observe is that both the lmtpds and the imapds are spending HUGE times waiting on locks. Even when the system load is only 1-2, an 'strace' attached to an instance of lmtpd or imapd shows waits
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
On Thu, 2008-02-28 at 16:56 -0600, Kenneth Marshall wrote: It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. Ken And if you can, try to get the mailstore over onto a RAID1. RAID5 is only good for long rebuilds and slow writes. Since you've already got DRBD setup, can you gank something to add as a second DRBD replica, which you can either test a single disk setup, or a RAID1 setup? Z Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
On Thu, 2008-02-28 at 18:36 -0500, Zachariah Mully wrote: On Thu, 2008-02-28 at 16:56 -0600, Kenneth Marshall wrote: It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. Ken And if you can, try to get the mailstore over onto a RAID1. RAID5 is only good for long rebuilds and slow writes. Since you've already got DRBD setup, can you gank something to add as a second DRBD replica, which you can either test a single disk setup, or a RAID1 setup? Err... I thought you could setup drbd in a multiple replica config, but I was mistaken... Z Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Gah my first thought was, a 3-disk RAID5? Is this 1998 or 2008? Disk is cheap. RAID-1 or RAID-10. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues
Jeff Fookson [EMAIL PROTECTED] wrote: Databases are all skiplist. As a rule of thumb, do not use skiplist for the duplicate delivery suppression database (deliver.db). Even if everybody hates it, use BerkeleyDB, Version 4.4.52 or higher. Give it a quite fair amount of shared memory. And run cyr_expunge often to prune that database so that no entry is older than - say - 3 days. We have approx 10-15 messages/sec incoming on one node. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
