Re: [Dovecot] Significant performance problems
On 10/6/10 11:26 PM, Brandon Davidson wrote: I do have one more idea I'll throw out there. Everything I've got here is virtual. I only have the one Dovecot/Postfix server running now, and the impression I get from you all is that that should be adequate for my load. What would the collective opinion be of simply removing the NFS server altogether and mounting the virtual disk holding my messages directly to the dovecot server? If you're not planning on doing some sort of HA failover or load balancing, and have the option to make your storage direct-attached instead of NAS, it might be worth trying. There's not much to be gained from NFS in a single-node configuration. I ended up implementing this just now. Still had load issues this morning and I'm hoping that removing NFS helps me out. Chris -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
Re: [Dovecot] Significant performance problems
For documentation's sake, here's what I've done so far: 1) Implemented Timo's fixes to my config file (fixed shared INDEX, adjusted nfs settings for reality of only one server hitting it) 2) installed imapproxy on the webmail server at the recommendation of the developers of that product (SOGo) 3) Modified my NFS mount with noatime to reduce i/o hits there. Need to figure out what Brad's suggestions about readahead on the server mean. 4) Threw gobs of RAM at both the dovecot server (went from 4GB to 8) and the NFS server (from 1GB to 8). Also cranked up vCPUs on each to 4. I hope that's enough to get things working much better tomorrow morning. I'll be back to report or beg for more. I really appreciate the quick responses and helpful advice. I do have one more idea I'll throw out there. Everything I've got here is virtual. I only have the one Dovecot/Postfix server running now, and the impression I get from you all is that that should be adequate for my load. What would the collective opinion be of simply removing the NFS server altogether and mounting the virtual disk holding my messages directly to the dovecot server? I give up the ability to have a failover dovecot/postfix server, which was my motivation for using NFS in the first place, but a usable system probably trumps a redundant one. Chris On 10/6/10 4:32 PM, Chris Hobbs wrote: Hi all, I'm sure my issues are a result of misconfiguration, but I'm hoping someone can point me in the right direction. I'm getting pressure to move us back to GroupWise, which I desperately want to avoid :-/ -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
Re: [Dovecot] Significant performance problems
On 10/6/10 6:28 PM, Rick Romero wrote: Is your disk a virtual disk as well? Have you checked performance? Something like: | hdparm -tT /dev/sda| On a ZFS RAID 10 of 10 7200RPM SATA drives, I get about 100MB/s At work we have an EMC SAN, and I get like 350MB/s on that beast - but if this is just a vhd on a Windows box, that could be an issue.. The disks are virtual on an EMC iSCSI san. I know you're supposed to run hdparm when things are quiet, but trying it now (somewhat quiet) I get: on the NFS server: Timing cached reads: 11844 MB in 2.00 seconds = 5928.81 MB/sec Timing buffered disk reads: 176 MB in 3.00 seconds = 58.62 MB/sec on the dovecot server (where indexes are stored): Timing cached reads: 12310 MB in 2.00 seconds = 6162.13 MB/sec Timing buffered disk reads: 230 MB in 3.02 seconds = 76.22 MB/sec -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
Re: [Dovecot] Significant performance problems
On 10/6/10 5:22 PM, Timo Sirainen wrote: Is the load CPU load or disk I/O load? If I/O load, what NFS operations are peaking there, or all of them? Pretty graphs of nfsstat output would be nice. If I'm reading the output of our monitoring system correctly, the CPU is spending quite a bit of time in WAIT status, so I asuume that means it is IO bound? After 19 minutes of uptime, nfsstat looks like (I'm not monitoring this [yet], so I don't have pretty graphs :/ ): Client nfs v3: null getattr setattr lookup access readlink 0 0% 2038915% 7615 5% 4219831% 2689620% 580% read writecreate mkdirsymlink mknod 6923 5% 5825 4% 4178 3% 290% 0 0% 0 0% remove rmdirrename link readdir readdirplus 2771 2% 0 0% 2500 1% 770% 7238 5% 1428 1% fsstat fsinfo pathconf commit 3 0% 4 0% 2 0% 5690 4% login_processes_count: 20 Probably could use less then 20. login_max_connections: 64 And this could be higher. In general you should have maybe 1-2x the number of login processes than CPU cores. Since this is in a virtual environment, I went ahead and ramped up the number of CPUs to 4, since I have the processors to spare. mail_nfs_storage: yes You said you have only one server accessing mails. So set this to "no". Done. mail_location: maildir:~/Maildir:INDEX=/var/indexes/%u .. namespace: type: shared separator: / prefix: shared/%%u/ location: maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u The INDEX path here is wrong now. Fixed - luckily most of our users don't share so this shouldn't have had a huge impact. Also you could try if maildir_very_dirty_syncs=yes is helpful. Done. Will report back tomorrow on how much these fixes help. Really appreciate the effort. -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
[Dovecot] Significant performance problems
Hi all, I'm sure my issues are a result of misconfiguration, but I'm hoping someone can point me in the right direction. I'm getting pressure to move us back to GroupWise, which I desperately want to avoid :-/ We're running dovecot 1.2.9 on Ubuntu 10.4 LTS+postfix. The server is a VM with 1 vCPU and 4GB of RAM. We serve about 10,000 users with anywhere from 500-1000 logged in at any one time. Messages are stored in Maildir format on two NFS servers (one for staff, the other for students). Today I implemented the "High performance" setup described here: http://wiki.dovecot.org/NFS (mainly moving indexes off of NFS, since I'm only using the one server). I also added imapproxy to our webmail client server (SOGo). The vast majority of our users come in over the web. We currently see load averages spiking into the 20-30 range. When this happens, service crawls to a near standstill, and ultimately the SOGo client starts crashing out. I'm wondering if anything jumps out at anybody here - feel free to mock if/when you find an obvious configuration problem. I just want it to work :-) dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-25-server x86_64 Ubuntu 10.04.1 LTS log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): *:2000 ssl_cert_file: /etc/dovecot/certs/mail_nhusd_k12_ca_us.crt ssl_key_file: /etc/dovecot/certs/mail_nhusd_k12_ca_us.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login login_process_per_connection: no login_process_size: 512 login_processes_count: 20 login_max_processes_count: 3000 login_max_connections: 64 max_mail_processes: 2048 mail_max_userip_connections(default): 20 mail_max_userip_connections(imap): 20 mail_max_userip_connections(pop3): 10 mail_max_userip_connections(managesieve): 10 mail_access_groups: staffmailusers mail_privileged_group: dovecot mail_uid: mail mail_gid: 502 mail_location: maildir:~/Maildir:INDEX=/var/indexes/%u mail_nfs_storage: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugins(default): acl imap_acl quota imap_quota expire mail_plugins(imap): acl imap_acl quota imap_quota expire mail_plugins(pop3): mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: shared separator: / prefix: shared/%%u/ location: maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u list: children lda: deliver_log_format: %$ -- FROM=%f SUBJECT=%s mail_plugins: cmusieve acl expire log_path: info_log_path: syslog_facility: mail postmaster_address:postmas...@nhusd.k12.ca.us hostname: mail.nhusd.k12.ca.us auth_socket_path: /var/run/dovecot/auth-master auth default: passdb: driver: pam passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 plugin: quota: maildir:User quota quota_rule: *:storage=9G quota_rule2: Trash:storage=200M acl: vfile acl_shared_dict:file:/home/staff/dovecot/shared-mailboxes expire: Trash 7 Trash/* 7 Spam 30 expire_dict: proxy::expire sieve: ~/.dovecot.sieve sieve_dir: ~/sieve sieve_extensions: +imapflags dict: expire: mysql:/etc/dovecot/dovecot-dict-expire.conf -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
Re: [Dovecot] EVERYONE USING DOVECOT PLEASE SIGN: Thanks, Administrators of Dovecot!
On 8/17/10 9:28 AM, Jerrale G wrote: *Our gratitude goes to, but not limited to:* *Timo Sirainen and Charles Marcus* Please add New Haven Unified School District to the chorus. We migrated away from GroupWise this summer and in addition to getting to use a better suite of products (dovecot+postfix+SOGo), we're saving the district a nice chunk of money. Thank you, -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
[Dovecot] Proxy Access (Manager/Secretary) Best Practices?
I've mostly got our dovecot+postfix+SOGo+openldap open source groupware replacement working the way I want it to; we're replacing GroupWise in our organization and I'm thrilled to be doing it. I'm supporting about 1,000 active staff users (and another 6,000 student accounts). I've got e-mail and calendar sharing working, and it does what it says it will do, but it is (go figure) different in concept from Proxy access under GroupWise. In GroupWise, I can give my secretary proxy access to my account and she can read all of my folders, see my calendars, and send e-mail as me. To someone that receives an e-mail or appointment request from her while she's proxied to me, there's no distinction at all. Is there a way to mimic this sort of functionality with the tools I've chosen? I've figured out that I can add additional 'mail' attributes to the secretary's record, and those addresses are available as drop-down choices in the SOGo web interface, but with the secretary's name and not the manager. Any advice will be much appreciated. And many thanks to the developers involved for writing and making available such amazingly good software. Eliminating the license fees we've been paying to Novell is allowing me to save a technician that surely would have been eliminated in budget cuts this year. Chris Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
Re: [Dovecot] Shared Mailbox Questions
Thanks for the reply Timo... >>> Timo Sirainen 06/11/10 7:07 AM >>> >> Jun 9 22:10:23 imap2 dovecot: dict: dict sql iterate failed: file is >> encrypted or is not a database > I don't know about this.. Is there more info I can provide to help troubleshoot this? -- This message was scanned by ESVA and is believed to be clean.
[Dovecot] Shared Mailbox Questions
Hi All,
I've pored over the wiki, read through the archive, and have what I believe
should be a working set-up, but I can't seem to get my shared namespace to work
correctly. Any guidance would be much appreciated.
I started off using a flat file for acl_shared_dict and ran into the same
problem recently reported with the shared-database file getting chown'd to the
first user to touch it with 0600 permissions. So I thought I'd give it a whirl
with sqlite.
Here are the (hopefully) relevant bits from dovecot -n (the entire output is
posted below). This is dovecot 1.2.9 running on a freshly installed ubuntu 10.4
server:
namespace:
type: shared
separator: /
prefix: shared/%%u/
location: maildir:/home/staff/%%n/Maildir:INDEX=~/Maildir/shared/%%u
list: children
plugin:
acl_shared_dict: proxy::acl
dict:
acl: sqlite:/etc/dovecot/acl-shared-dict.conf
acl-shared-dict.conf looks like this:
connect = /home/staff/dovecot/acl-shared-ns.sqlite
map {
table = acl_shared_ns
pattern = shared/shared-boxes/user/$to/$from
value_field = has_visible_folders
fields {
from_user = $from
to_user = $to
}
}
/home/staff/dovecot/acl-shared-ns.sqlite is set 0666, and I can add records to
it from the sqlite command line as a normal user. The table was created with:
CREATE TABLE acl_shared_ns (from_user, to_user, has_visible_folders, PRIMARY
KEY (from_user) ON CONFLICT REPLACE);
When I attempt to SETACL from a telnet imap session, I get the following errors
in mail.err:
Jun 9 22:10:23 imap2 dovecot: dict: dict sql iterate failed: file is encrypted
or is not a database
Jun 9 22:10:23 imap2 dovecot: dict: dict sql iterate failed: file is encrypted
or is not a database
Jun 9 22:10:23 imap2 dovecot: dict: sqlite: exec(INSERT INTO acl_shared_ns
(has_visible_folders,to_user,from_user) VALUES ('1','chobbs','IOchoa')) failed:
file is encrypted or is not a database (26)
Jun 9 22:10:23 imap2 dovecot: IMAP(IOchoa): read(/var/run/dovecot/dict-server)
failed: Remote disconnected
Jun 9 22:10:23 imap2 dovecot: IMAP(IOchoa): acl: dict commit failed
Jun 9 22:10:23 imap2 dovecot: dovecot: child 8268 (dict) killed with signal 11
(core dumps disabled)
Running that same "INSERT INTO..." as the IOchoa user from the sqlite command
line works just fine.
At this point any further deleteacl or setacl commands simply hang, presumably
because there's no longer a running dict process.
Here is the entire output of dovecot -n:
# 1.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-22-server x86_64 Ubuntu 10.04 LTS
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap imaps pop3 pop3s managesieve
listen(default): *
listen(imap): *
listen(pop3): *
listen(managesieve): *:2000
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
login_executable(managesieve): /usr/lib/dovecot/managesieve-login
mail_privileged_group: mail
mail_uid: mail
mail_gid: mail
mail_location: maildir:~/Maildir
mmap_disable: yes
mail_nfs_storage: yes
mail_nfs_index: yes
mbox_write_locks: fcntl dotlock
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_executable(managesieve): /usr/lib/dovecot/managesieve
mail_plugins(default): acl imap_acl
mail_plugins(imap): acl imap_acl
mail_plugins(pop3):
mail_plugins(managesieve):
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve
namespace:
type: private
separator: /
inbox: yes
list: yes
subscriptions: yes
namespace:
type: shared
separator: /
prefix: shared/%%u/
location: maildir:/home/staff/%%n/Maildir:INDEX=~/Maildir/shared/%%u
list: children
lda:
deliver_log_format: %$ -- FROM=%f SUBJECT=%s
mail_plugins: cmusieve acl
log_path:
info_log_path:
syslog_facility: mail
postmaster_address: postmas...@nhusd.k12.ca.us
hostname: mail.nhusd.k12.ca.us
auth default:
passdb:
driver: pam
passdb:
driver: ldap
args: /etc/dovecot/dovecot-ldap.conf
userdb:
driver: passwd
userdb:
driver: ldap
args: /etc/dovecot/dovecot-ldap.conf
plugin:
acl: vfile
acl_shared_dict: proxy::acl
sieve: ~/.dovecot.sieve
sieve_dir: ~/sieve
dict:
acl: sqlite:/etc/dovecot/acl-shared-dict.conf
Chris Hobbs
Director, Technology
New Haven Unified School District
--
This message was scanned by ESVA and is believed to be clean.
