Re: [Dovecot] Indexes to MLC-SSD
Reasons to choose ZFS were snapshots, and mainly dedup and compression capabilities. I know, it's ironic since I'm not able to use them now due to severe performance issues with them (mostly dedup) turned on. I do like the emphasis on data integrity and fast on-the-fly configurability of ZFS to an extent, but I wouldn't recommend it highly for new users, especially for production. It works (in fact it's working right now), but has its fair share of troubles. We've started implementations to move our mail system to a more modular enviroment and we'll probably move away from ZFS. Was a nice experiment nonetheless, I learned quite a bit from it. On Thu, Nov 3, 2011 at 12:27, Ed W li...@wildgooses.com wrote: On 03/11/2011 11:32, Felipe Scarel wrote: I'm using native ZFS (http://zfsonlinux.org) on production here (15k+ users, over 2TB of mail data) with little issues. Dedup and compression disabled, mind that. OT: but what were the rough criteria that led you to using ZFS over say LVM with EXT4/XFS/btrfs? I can think of plenty for/against reasons for each, just wondering what criteria affected *your* situation? I'm guessing some kind of manageability reason is at the core, but perhaps you can expand on how it's all worked out for you? I have a fairly static server setup here so I have been satisfied with LVM, software raid and mainly ext4. The main thing I miss is simple to use snapshots Cheers Ed W
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Quick question about the usage of DRBD: I'm thinking of a setup on my organization here (15k+ users, 4TB of email data), but I'm holding back on the clusterization due to the high volume of data. Using DRBD would implicate mirroring those 4TB of data across all cluster nodes? If yes, I might go with a SAN-based solution, though I haven't studied much about that setup yet (the other sysadm administrates the VMs and SAN, gotta ask him a few questions). On Mon, Oct 31, 2011 at 08:00, Robert Schetterer rob...@schetterer.orgwrote: Am 31.10.2011 10:43, schrieb Arlin: Hi Robert, Thanks for the reply. We are upgrading Dovecot to v2.0.15 and other component's to the latest version. In that case, can we use san for storage or are you recommending that drbd with ocfs2 is the best way to attain the horizontal scalability for the mail storage? Hi Arlin, there is no best way, you should choose whatever fits best to your needs an haves so it depends on many stuff ( i.e at last finance, network, manpower, knowledge) etc So all i can say iam just using a loadbalanced cluster setup with drbd ocfs2 maildir dovecot postfix mysql clamav spamassassin on ubuntu lucid lts with 3000 Mailboxes without any big Problems yet but i can imagine that a professional SAN might be better in performance but there is a lot other other questions left , i.e maildir must not be the best solution for mailbox format etc cluster setups with lots of mailboxes are complex in many ways, if you planning a real big mailservice you should ask more here on this list for existing other setups and choose i.e Timo and/or others for professional and paid advice and work Thanks, Arlin -Original Message- From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] On Behalf Of Robert Schetterer Sent: 31 October 2011 14:26 To: dovecot@dovecot.org Subject: Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers? Am 31.10.2011 09:47, schrieb Arlin: Could anyone please respond to this query. Thank you! you may use loadbalancers i.e (keepalived etc) and/or http://wiki2.dovecot.org/FeatLoginProxy http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy look the list for cluster setups etc reading your former post you want to use many outdated prog versions dont do that a san for storage might be a good choice some of the list use drbd with ocfs2 and other cluster filesystems From: Arlin [mailto:ar...@mvs.us] Sent: 28 October 2011 17:06 To: 'dovecot@dovecot.org' Subject: How can we horizontally scale Dovecot across multiple servers? Hi, How can we horizontally scale Dovecot across multiple servers? Do we require to install independent instances of Dovecot on each server? We are planning to use a NAS/SAN device using ZFS or EFS for email storage. Each logical unit will be of 10TB and similarly as the no: of user increases we are planning to add multiple 10TB units. In this case how we can manage the email storage on multiple volumes from Dovecot. The configuration of our existing system is:- Dovecot 1.0.15 / Maildirs Postfix 2.5.5 Debian 5.0.9 (Lenny) MySQL 5.0.15 Please advise. Thanks in advance. Creative Regards, Arlin -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] limiting number of incorrect logins per connection
Alex, I've not personally done it (so just speculating here, bear with me) but you can customize Fail2Ban's actions if needed. So, if you can match the attemps through some regex (and since you're seeing them in the logs, that should be quite possible), then you can edit one of the 'actions' to drop the connection for ip. I'm just not entirely sure that iptables (or pf, or whatever firewall you've got) can do it to active connections, 'cause that problem hasn't arised for me so far. On Fri, Aug 26, 2011 at 06:14, Alex a...@ahhyes.net wrote: I am happy to recompile if there is no config option. I gather it's in the src/auth dir somewhere in one of the C source files. Just need to be pointed in the right dir. On Fri, 26 Aug 2011 19:07:08 +1000, Alex wrote: 3 minutes! I think that's too long, how can I drop that down to about 45 seconds? On Fri, 26 Aug 2011 11:44:45 +0300, Timo Sirainen wrote: On 26.8.2011, at 10.25, Alex wrote: Running Dovecot 2 on my server. It is regularly getting dictionary auth attacked. What I have noticed is that once connected to a pop3/imap login session, you can send endless incorrect usernames+passwords attempts. This is a problem for me... I use fail2ban to try and stop these script kiddies. The problem is that fail2ban detects the bad auths, firewalls the IP, however, since it's an established session, the attacker can keep authing away... It's only on a subsequent (new) connection that the firewalling will take effect. Umm. If client hasn't managed to log in in 3 minutes, it's disconnected (no matter what it does with the connection).
Re: [Dovecot] limiting number of incorrect logins per connection
Yeah, I had read about half of that thread, and after I sent my mail kept reading and stumbled upon this: (...) using the recent module needs dovecotto close the connection upon authentication failure, as iptables only (normally) comes in to play for new connections (...). So, yeah, my suggestion probably won't work. On Fri, Aug 26, 2011 at 09:15, Felipe Scarel fbsca...@gmail.com wrote: Alex, I've not personally done it (so just speculating here, bear with me) but you can customize Fail2Ban's actions if needed. So, if you can match the attemps through some regex (and since you're seeing them in the logs, that should be quite possible), then you can edit one of the 'actions' to drop the connection for ip. I'm just not entirely sure that iptables (or pf, or whatever firewall you've got) can do it to active connections, 'cause that problem hasn't arised for me so far. On Fri, Aug 26, 2011 at 06:14, Alex a...@ahhyes.net wrote: I am happy to recompile if there is no config option. I gather it's in the src/auth dir somewhere in one of the C source files. Just need to be pointed in the right dir. On Fri, 26 Aug 2011 19:07:08 +1000, Alex wrote: 3 minutes! I think that's too long, how can I drop that down to about 45 seconds? On Fri, 26 Aug 2011 11:44:45 +0300, Timo Sirainen wrote: On 26.8.2011, at 10.25, Alex wrote: Running Dovecot 2 on my server. It is regularly getting dictionary auth attacked. What I have noticed is that once connected to a pop3/imap login session, you can send endless incorrect usernames+passwords attempts. This is a problem for me... I use fail2ban to try and stop these script kiddies. The problem is that fail2ban detects the bad auths, firewalls the IP, however, since it's an established session, the attacker can keep authing away... It's only on a subsequent (new) connection that the firewalling will take effect. Umm. If client hasn't managed to log in in 3 minutes, it's disconnected (no matter what it does with the connection).
[Dovecot] Sharing all mailboxes and userdb LDAP attrs
Hello all, I'm setting up a Dovecot environment here, version 1.2.15 on Debian 6.0.2 squeeze. This is actually a complete revamp of the previous setup we have in-place here, built from the ground up with updated versions of all involved software. The operators have told me that they use some scripts hacked up by a previous sysadmin to give a single admin account full access to all user mail. That is, if any user runs into problems, they: 1. Call in; 2. The operator logs in as the admin user; 3. Operator performs maintenance duties on user email. I've been researching the possibility of using Dovecot shared namespaces to perform that very same task in a better fashion in this new server. So far, I've been able to globally share users' INBOXes and view them from a single admin account (through user= entries on global acl's). My ultimate goal, however, is to have access to all user mailboxes with any user that's a member of a particular group, adding all operators to that group as needed. - - - - - First question, then, is this one: how can I give global access to all user mailboxes? I've read that it's possible to give access to all subfolders of a particular folder throught the use of a .DEFAUL acl. That didn't seem to work with the uppermost directory, however. Here's what I tried: root@mail:/etc/dovecot# dovecot -a | grep acl: acl: vfile:/etc/dovecot/acl:cache_secs=300 root@mail:/etc/dovecot# cat acl/.DEFAULT owner lrwstipekxa user=admin lrwstipekxa Renaming .DEFAULT to INBOX does achieve the intended goal, but only for the INBOX folder evidently. - - - - - Second question is somewhat simpler. So far I've been using a single admin user, but I'd like to switch to using an admin group in the future. I've read that the best way to do that would be to use the user_attrs entry in my dovecot-ldap.conf file, while using a userdb ldap. The groups should be strings separated by commas in the appropriate attribute, from what I understand. Is there any readily-available or recommended schema I can use to fill up that attribute? I'm using the default ones (plus samba.schema) but I've seen mostly space to fit GID's, not group names. Thanks in advance, fbscarel PS: Here's my dovecot -a output, should it be needed. - - - - - root@mailaluno:~# dovecot -a # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 base_dir: /var/run/dovecot log_path: /var/log/dovecot/error.log info_log_path: /var/log/dovecot/info.log log_timestamp: %Y-%m-%d %H:%M:%S syslog_facility: mail protocols: imap pop3 pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): localhost:2000 ssl_listen: 127.0.0.1 ssl: yes ssl_ca_file: ssl_cert_file: /etc/ssl/certs/dovecot.pem ssl_key_file: /etc/ssl/private/dovecot.pem ssl_key_password: ssl_parameters_regenerate: 168 ssl_cipher_list: ssl_cert_username_field: commonName ssl_verify_client_cert: no disable_plaintext_auth: no verbose_ssl: yes shutdown_clients: yes nfs_check: yes version_ignore: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login login_user: dovecot login_greeting: Server ready. login_log_format_elements: user=%u method=%m rip=%r lip=%l %c login_log_format: %$: %s login_process_per_connection: no login_chroot: yes login_trusted_networks: login_process_size: 64 login_processes_count: 5 login_max_processes_count: 128 login_max_connections: 256 valid_chroot_dirs: mail_chroot: max_mail_processes: 512 mail_max_userip_connections: 10 verbose_proctitle: no first_valid_uid: 108 last_valid_uid: 0 first_valid_gid: 112 last_valid_gid: 0 mail_access_groups: mail_privileged_group: mail mail_uid: mail_gid: mail_location: mail_cache_fields: mail_never_cache_fields: imap.envelope mail_cache_min_mail_count: 0 mailbox_idle_check_interval: 30 mail_debug: yes mail_full_filesystem_access: no mail_max_keyword_length: 50 mail_save_crlf: no mmap_disable: no dotlock_use_excl: yes fsync_disable: no mail_nfs_storage: no mail_nfs_index: no mailbox_list_index_disable: yes lock_method: fcntl maildir_stat_dirs: no maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: no maildir_very_dirty_syncs: no mbox_read_locks: fcntl mbox_write_locks: fcntl dotlock mbox_lock_timeout: 300 mbox_dotlock_change_timeout: 120 mbox_min_index_size: 0 mbox_dirty_syncs: yes mbox_very_dirty_syncs: no mbox_lazy_writes: yes dbox_rotate_size: 2048 dbox_rotate_min_size: 16 dbox_rotate_days: 1 mail_drop_priv_before_exec: no mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_process_size: 256 mail_plugins(default): quota imap_quota trash mail_log acl imap_acl mail_plugins(imap): quota imap_quota trash mail_log acl
Re: [Dovecot] Sharing all mailboxes and userdb LDAP attrs
You know when you ask that stupid question and then realize you had it all along? Duh... And to top it off, I HAVE configured a master user on my Dovecot install and wasn't using it... man, do I feel stupid now! :) Thanks a bunch Charles! On Fri, Aug 19, 2011 at 13:44, Charles Marcus cmar...@media-brokers.comwrote: On 2011-08-19 12:14 PM, Felipe Scarel fbsca...@gmail.com wrote: I'm setting up a Dovecot environment here, version 1.2.15 on Debian 6.0.2 squeeze. This is actually a complete revamp of the previous setup we have in-place here, built from the ground up with updated versions of all involved software. The operators have told me that they use some scripts hacked up by a previous sysadmin to give a single admin account full access to all user mail. That is, if any user runs into problems, they: 1. Call in; 2. The operator logs in as the admin user; 3. Operator performs maintenance duties on user email. Isn't this what master users are for? http://wiki2.dovecot.org/Authentication/MasterUsers -- Best regards, Charles
Re: [Dovecot] mail spool filesystem
I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far. They don't advertise it as exactly production quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load. On Fri, Aug 19, 2011 at 14:49, Seth Mattinen se...@rollernet.us wrote: On 8/17/11 7:42 AM, Adrian Ulrich wrote: I read that XFS is a good choice, but is not too reliable... Are you using Maildir or MBOX? In any case: XFS would be my last choice: XFS is nice if you are working with large files ( 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files. I'd have to disagree. This is completely anecdotal, but I originally deployed ext3 on all of my mail servers (Dovecot maildir) and spools (Postfix) until they started exhibiting loading issues when busy. Reformatting into XFS resolved the problem with no other changes. I didn't have time to do any comparisons or gather statistics since it was an emergency situation and this was before ext4, but XFS has performed flawlessly for me. ~Seth
Re: [Dovecot] mail spool filesystem
I was not aware of that... I went with FUSE to test the deduplication feature of ZFS. I'll check out this link you've provided, many thanks Dave. :) On Fri, Aug 19, 2011 at 15:48, Dave McGuire mcgu...@neurotica.com wrote: On 08/19/2011 02:45 PM, Felipe Scarel wrote: I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far. They don't advertise it as exactly production quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load. You are aware that there's a real in-kernel ZFS implementation under Linux now, right? See http://zfsonlinux.org/. I've done some very basic testing with it, and so far, it works. Going through FUSE is slower than pissing tar; this implementation won't have that problem. FUSE is useful for many things. Performance-sensitive filesystems on production servers is oh-so-NOT one of them. ;) -Dave -- Dave McGuire Port Charlotte, FL
Re: [Dovecot] mail spool filesystem
Thanks, I've read some of the FAQ and install instructions and it seems pretty straightforward... I wish I could use Solaris but we're virtualizing everything on our Dell blade through VMWare ESXi and it's somewhat of a company policy to use the template Debian that's maintained by the senior sysadmin. About the compression, I've read some benchmarks/tests and the default lzjb algorithm seems to be a good cost/benefit for the usual applications. Without many reads to the filesystem, gzip compresses a whole lot better tho. On Fri, Aug 19, 2011 at 16:01, Dave McGuire mcgu...@neurotica.com wrote: Good luck! FYI, my mail spools are on ZFS filesystems under Solaris on UltraSPARC. It is lightning fast with 100+ dovecot imap processes pounding away. I've not yet enabled compression and done the copy/recopy dance, though. -Dave On 08/19/2011 02:57 PM, Felipe Scarel wrote: I was not aware of that... I went with FUSE to test the deduplication feature of ZFS. I'll check out this link you've provided, many thanks Dave. :) On Fri, Aug 19, 2011 at 15:48, Dave McGuiremcgu...@neurotica.com wrote: On 08/19/2011 02:45 PM, Felipe Scarel wrote: I'm testing out ZFS-fuse on my new install (talked about it on the other thread), no issues so far. The builtin deduplication and compression sure do help a lot, roughly 30% less storage space required so far. They don't advertise it as exactly production quality, but I'm willing to try it out, we're doing regular backups. The mail system hasn't gone live yet though, so I'm a bit uneasy on the performance side of things under heavy load. You are aware that there's a real in-kernel ZFS implementation under Linux now, right? See http://zfsonlinux.org/. I've done some very basic testing with it, and so far, it works. Going through FUSE is slower than pissing tar; this implementation won't have that problem. FUSE is useful for many things. Performance-sensitive filesystems on production servers is oh-so-NOT one of them. ;) -Dave -- Dave McGuire Port Charlotte, FL -- Dave McGuire Port Charlotte, FL
