Re: [Dovecot] Maildir Synchronization warnings
On 5.8.2013, at 20.22, /dev/rob0 wrote: >> To improve performance you can remove these two since you're using >> director. Also you could set maildir_very_dirty_syncs=yes. > > What about mail_fsync=always and mmap_disable=yes? These are needed > for non-director NFS, but what about with director? I'm not sure if mail_fsync=always is strictly required, but I don't think it's going to give you any performance improvements in any case. With mail_fsync=always the file is flushed to NFS storage in the fsync() call, while with mail_fsync=never the file is flushed to NFS storage in close() call. In both cases it's going to be flushed, but in error cases it's more confusing to see close() fail. mmap_disable=yes isn't required, but if the NFS storage dies the processes start dying with SIGBUS instead of logging "read() failed: Input/output error". I'm not entirely sure how changing it affects performance. Could be worse or could be better. If someone finds our let me know. :)
Re: [Dovecot] Maildir Synchronization warnings
On Fri, Aug 02, 2013 at 03:37:04PM +0300, Timo Sirainen wrote: > On Fri, 2013-08-02 at 20:21 +0800, Kavish Karkera wrote: > > > We have 2 pop/imap servers running with director. > > > > Dovecot version = 2.1.12 > > Dovecot version = 2.1.13 > .. > > mail_nfs_index = yes > > mail_nfs_storage = yes > > To improve performance you can remove these two since you're using > director. Also you could set maildir_very_dirty_syncs=yes. What about mail_fsync=always and mmap_disable=yes? These are needed for non-director NFS, but what about with director? -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Re: [Dovecot] Maildir Synchronization warnings
On 8/2/2013 6:57 AM, Timo Sirainen wrote: > On Fri, 2013-08-02 at 15:34 +0800, Kavish Karkera wrote: >> Hi, >> >> We are repeatedly getting these below warnings for some of our users, al >> though we have no complaints from them yet, >> we need to know why these warning occurs. >> >> So it would be help full if some one explain these warning msg in detail. > .. >> Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: >> Maildir: Scanning >> /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 94 >> seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1) > > It means that the maildir INBOX is huge, and it takes a long time to > access them with your available disk IO. Possibilities: Yes, 23K+ and 43K+ emails in a maildir INBOX will generate quite a bit of read IO. Are these shared mailboxes used for some business process, or normal user mailboxes? If the former are these tends of thousands of emails arriving in a single day? > a) Move move of your mails away from INBOX. If the former you'll want to create a workflow that moves each email to archival storage, or some temporary location, after each email has been processed, either by your automated system or human beings. As you've discovered, allowing so many emails to pile up in a maildir INBOX causes problems. If the latter, the users need to be properly educated on mail folder hierarchy, sorting, and storage best practices. > b) Switch to different mailbox format that can handle large mailboxes, > such as mdbox or sdbox. This will help but not nearly as much as Timo's first suggestion. For many reasons it's not always possible or feasible for an organization to switch mailbox storage formats. So the last option, which is always the least bang for the buck, is optimizing your storage stack on the NFS server for maximum IOPS. For instance if you're currently using parity RAID, migrating to RAID10 will give you a 5-15x+ boost in mixed read/write random IOPS throughput. If your RAID has large strips (chunks) of say 128KB+ switching to small strips of 16-32KB will increase IOPS, especially if you're using parity RAID--smaller chunks significantly decrease RMW latency. Switching from EXT3/4 to XFS w/inode64 will provide a boost due to parallelism across allocation groups, and files being clustered around their metadata inodes which decreases head seek latency--this is especially beneficial to maildir which manipulates metadata more than file data. And then there's always the option of adding more hardware, typically more/faster spindles, maybe more HW RAID cache and more system RAM for greater filesystem buffer cache, or all three. -- Stan
Re: [Dovecot] Maildir Synchronization warnings
On 8/2/2013 8:21 AM, Kavish Karkera wrote:
Sorry Charles, it was posted twice unintentionally (becuase of network issue).
No worries... now, please don't top-post... ;)
We have 2 pop/imap servers running with director.
Dovecot version = 2.1.12
Dovecot version = 2.1.13
Very first thing on your troubleshooting ToDo list should always be to
make sure you are using the latest version of whichever series you
decide to run.
So, you should upgrade to 2.1.17
Output of doveconf -n
# 2.1.12: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
If you aren't willing to do that for political reasons (ie, phbfh) to do
with the distro you are using, then your first line of support should be
that distro's support service, because they may or may not have
incorporated certain bug fixes in the older version...
default_client_limit = 5
default_vsz_limit = 1 G
disable_plaintext_auth = no
doveadm_proxy_port = 24245
first_valid_uid = 99
last_valid_gid = 99
listen = *
mail_fsync = always
mail_location = maildir:~/Maildir:INDEX=/indexes/%h:CONTROL=/indexes/%h
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = quota
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
quota = maildir:storage=1024
quota_rule = *:storage=1024
}
service auth {
client_limit = 5
unix_listener auth-userdb {
group = nobody
mode = 0666
user = nobody
}
}
service doveadm {
inet_listener {
port = 24245
}
}
service imap-login {
process_min_avail = 8
service_count = 0
vsz_limit = 1 G
}
service imap {
process_limit = 4096
service_count = 0
vsz_limit = 1 G
}
service pop3-login {
process_min_avail = 8
service_count = 0
vsz_limit = 1 G
}
service pop3 {
process_min_avail = 8
service_count = 0
vsz_limit = 1 G
}
ssl_cert =
Re: [Dovecot] Maildir Synchronization warnings
On Fri, 2013-08-02 at 20:21 +0800, Kavish Karkera wrote: > We have 2 pop/imap servers running with director. > > Dovecot version = 2.1.12 > Dovecot version = 2.1.13 .. > mail_nfs_index = yes > mail_nfs_storage = yes To improve performance you can remove these two since you're using director. Also you could set maildir_very_dirty_syncs=yes.
Re: [Dovecot] Maildir Synchronization warnings
Sorry Charles, it was posted twice unintentionally (becuase of network issue).
We have 2 pop/imap servers running with director.
Dovecot version = 2.1.12
Dovecot version = 2.1.13
Output of doveconf -n
# 2.1.12: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
default_client_limit = 5
default_vsz_limit = 1 G
disable_plaintext_auth = no
doveadm_proxy_port = 24245
first_valid_uid = 99
last_valid_gid = 99
listen = *
mail_fsync = always
mail_location = maildir:~/Maildir:INDEX=/indexes/%h:CONTROL=/indexes/%h
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = quota
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
quota = maildir:storage=1024
quota_rule = *:storage=1024
}
service auth {
client_limit = 5
unix_listener auth-userdb {
group = nobody
mode = 0666
user = nobody
}
}
service doveadm {
inet_listener {
port = 24245
}
}
service imap-login {
process_min_avail = 8
service_count = 0
vsz_limit = 1 G
}
service imap {
process_limit = 4096
service_count = 0
vsz_limit = 1 G
}
service pop3-login {
process_min_avail = 8
service_count = 0
vsz_limit = 1 G
}
service pop3 {
process_min_avail = 8
service_count = 0
vsz_limit = 1 G
}
ssl_cert =
To: dovecot@dovecot.org
Sent: Friday, 2 August 2013 4:14 PM
Subject: Re: [Dovecot] Maildir Synchronization warnings
On 2013-08-02 4:36 AM, Kavish Karkera wrote:
> We are repeatedly getting these below warnings for some of our users, al
> though we have no complaints from them yet, we need to know why these warning
> occurs.
>
> So it would be help full if some one explain these warning msg in detail.
>
>
> Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning:
> Maildir:
> Scanning/mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took
> 94 seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1)
Please don't spam the list with multiple postings of the same question.
Most people do not have crystal balls, and those that do often find them
broken, so it would be helpful if you could provide at least a bare
minimum of details about your setup...
What version of dovecot is this? doveconf -n output? Singe server or
cluster?
--
Best regards,
Charles
Re: [Dovecot] Maildir Synchronization warnings
Thanks Timo, Temporarly would move the messages and keep a watch. Updating to mdbox is add to the list. Thanks&Regards, Kavish Karkera From: Timo Sirainen To: Kavish Karkera Cc: "dovecot@dovecot.org" Sent: Friday, 2 August 2013 5:27 PM Subject: Re: [Dovecot] Maildir Synchronization warnings On Fri, 2013-08-02 at 15:34 +0800, Kavish Karkera wrote: > Hi, > > We are repeatedly getting these below warnings for some of our users, al > though we have no complaints from them yet, > we need to know why these warning occurs. > > So it would be help full if some one explain these warning msg in detail. .. > Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: > Maildir: Scanning > /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 94 > seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1) It means that the maildir INBOX is huge, and it takes a long time to access them with your available disk IO. Possibilities: a) Move move of your mails away from INBOX. b) Switch to different mailbox format that can handle large mailboxes, such as mdbox or sdbox.
Re: [Dovecot] Maildir Synchronization warnings
On Fri, 2013-08-02 at 15:34 +0800, Kavish Karkera wrote: > Hi, > > We are repeatedly getting these below warnings for some of our users, al > though we have no complaints from them yet, > we need to know why these warning occurs. > > So it would be help full if some one explain these warning msg in detail. .. > Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: > Maildir: Scanning > /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 94 > seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1) It means that the maildir INBOX is huge, and it takes a long time to access them with your available disk IO. Possibilities: a) Move move of your mails away from INBOX. b) Switch to different mailbox format that can handle large mailboxes, such as mdbox or sdbox.
Re: [Dovecot] Maildir Synchronization warnings
On 2013-08-02 4:36 AM, Kavish Karkera wrote: We are repeatedly getting these below warnings for some of our users, al though we have no complaints from them yet, we need to know why these warning occurs. So it would be help full if some one explain these warning msg in detail. Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning/mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 94 seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1) Please don't spam the list with multiple postings of the same question. Most people do not have crystal balls, and those that do often find them broken, so it would be helpful if you could provide at least a bare minimum of details about your setup... What version of dovecot is this? doveconf -n output? Singe server or cluster? -- Best regards, Charles
[Dovecot] Maildir Synchronization warnings
Hi, We are repeatedly getting these below warnings for some of our users, al though we have no complaints from them yet, we need to know why these warning occurs. So it would be help full if some one explain these warning msg in detail. - Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 94 seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1) Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: Maildir /mail/v3store/example.com/kavish.kark...@example.com/Maildir: Synchronization took 94 seconds (0 new msgs, 0 flag change attempts, 0 expunge attempts) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 154 seconds (43129 readdir()s, 0 rename()s to cur/, why=0x1) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 162 seconds (43129 readdir()s, 0 rename()s to cur/, why=0xc) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Maildir /mail/v3store/example.com/kavish.kark...@example.com/Maildir: Synchronization took 162 seconds (0 new msgs, 0 flag change attempts, 0 expunge attempts) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Locking transaction log file /indexes//mail/v3store/example.com/kavish.kark...@example.com/.INBOX/dovecot.index.log took 50 seconds - Thanks In advance, Regards, Kavish Karkera
[Dovecot] Maildir Synchronization warnings
Hi, We are repeatedly getting these below warnings for some of our users, al though we have no complaints from them yet, we need to know why these warning occurs. So it would be help full if some one explain these warning msg in detail. - Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 94 seconds (23191 readdir()s, 0 rename()s to cur/, why=0x1) Aug 2 12:52:55 blade8 dovecot: imap(kavish.kark...@example.com): Warning: Maildir /mail/v3store/example.com/kavish.kark...@example.com/Maildir: Synchronization took 94 seconds (0 new msgs, 0 flag change attempts, 0 expunge attempts) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 154 seconds (43129 readdir()s, 0 rename()s to cur/, why=0x1) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Maildir: Scanning /mail/v3store/example.com/kavish.kark...@example.com/Maildir/cur took 162 seconds (43129 readdir()s, 0 rename()s to cur/, why=0xc) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Maildir /mail/v3store/example.com/kavish.kark...@example.com/Maildir: Synchronization took 162 seconds (0 new msgs, 0 flag change attempts, 0 expunge attempts) Aug 2 12:53:54 blade6 dovecot: imap(kavish.kark...@example.com): Warning: Locking transaction log file /indexes//mail/v3store/example.com/kavish.kark...@example.com/.INBOX/dovecot.index.log took 50 seconds - Thanks In advance, Regards, Kavish Karkera
