Re: Sis to deduplicate attachments does not work?
Thanks for your sharing, Daniel. On Wed, Apr 24, 2019 at 4:12 PM Daniel Miller wrote: > On April 23, 2019 10:54:38 PM luckydog xf wrote: > >> Is it worthwile to use dbox? seeing from >> http://www.linuxmail.info/mbox-maildir-mail-storage-formats/ it may >> cause file lock and easy to corrupt. >> >> As with everything - it depends. You're asking me so these are *my* > opinions - and I do not claim to be anything more than a hobbyist/tinkerer > when the comes to this. > > mbox has potential use for long term read-only archives - I see no reason > to use it for live mailboxes. > > maildir is undoubtedly the least susceptible to corruption. It's also the > slowest format for reading. How slow is "slow" depends on your hardware - > it may be imperceptible with enough RAM and SSD's - or it may result in > user complaints with large mailboxes. > > dbox is Dovecot's preferred format. I know Timo has put a lot of effort > into it. sdbox is similar to maildir in that each mail is a separate file. > mdbox significantly reduces the number of files which can make file-based > backups faster. Both dbox formats are dependent on their index files. > > If you've got good hardware, including a proper UPS, I'd recommend dbox > (my server is presently using sdbox). With large mailboxes and file-based > backups you'll benefit from mdbox. When reliability is the #1 concern above > anything else - use maildir. Depending on your use SIS can have significant > impact on storage requirements - but storage these days is relatively cheap. > > I haven't seen much feedback from users actively using SIS - I'd love to > hear from high traffic sites with SIS experience to know if the corruption > issues have been resolved. In my case there was at least a 30% reduction in > space but I had too many errors - admittedly it's been a couple years since > I last tried it. > > -- > Daniel >
Re: Sis to deduplicate attachments does not work?
It seems Maildir is the safest mail format right now, as long as you could accept little performance and disk cost penalty. On Wed, Apr 24, 2019 at 9:01 PM Tanstaafl via dovecot wrote: > On Wed Apr 24 2019 04:12:30 GMT-0400 (Eastern Standard Time), Daniel > Miller via dovecot wrote: > > If you've got good hardware, including a proper UPS, I'd recommend dbox > > (my server is presently using sdbox). With large mailboxes and > > file-based backups you'll benefit from mdbox. When reliability is the #1 > > concern above anything else - use maildir. Depending on your use SIS can > > have significant impact on storage requirements - but storage these days > > is relatively cheap. > > My plan when I roll out my new server this year is to use mdbox, but put > the indexes and other important meta data on a smallish volume using > either ZFS or BTRFS, for the automatic self-healing capabilities (and > the ability to expand it if necessary). > > This pretty much eliminates the worry about data loss from index file > corruption. > > > I haven't seen much feedback from users actively using SIS - I'd love to > > hear from high traffic sites with SIS experience to know if the > > corruption issues have been resolved. In my case there was at least a > > 30% reduction in space but I had too many errors - admittedly it's been > > a couple years since I last tried it. > > I never tried it because of the problems with respect to backup/restore, > and if I'm not mistaken, those problems have not been resolved. > > Maybe its a design issue... > > Or maybe it just isn't a high enough priority, like the missing > x-original-to header in the LMTP code that will still prevent me from > being able to use the otherwise much better LMTP delivery agent. >
Re: Sis to deduplicate attachments does not work?
On Wed Apr 24 2019 04:12:30 GMT-0400 (Eastern Standard Time), Daniel Miller via dovecot wrote: > If you've got good hardware, including a proper UPS, I'd recommend dbox > (my server is presently using sdbox). With large mailboxes and > file-based backups you'll benefit from mdbox. When reliability is the #1 > concern above anything else - use maildir. Depending on your use SIS can > have significant impact on storage requirements - but storage these days > is relatively cheap. My plan when I roll out my new server this year is to use mdbox, but put the indexes and other important meta data on a smallish volume using either ZFS or BTRFS, for the automatic self-healing capabilities (and the ability to expand it if necessary). This pretty much eliminates the worry about data loss from index file corruption. > I haven't seen much feedback from users actively using SIS - I'd love to > hear from high traffic sites with SIS experience to know if the > corruption issues have been resolved. In my case there was at least a > 30% reduction in space but I had too many errors - admittedly it's been > a couple years since I last tried it. I never tried it because of the problems with respect to backup/restore, and if I'm not mistaken, those problems have not been resolved. Maybe its a design issue... Or maybe it just isn't a high enough priority, like the missing x-original-to header in the LMTP code that will still prevent me from being able to use the otherwise much better LMTP delivery agent.
Re: Sis to deduplicate attachments does not work?
On April 23, 2019 10:54:38 PM luckydog xf wrote: Is it worthwile to use dbox? seeing from http://www.linuxmail.info/mbox-maildir-mail-storage-formats/ it may cause file lock and easy to corrupt. As with everything - it depends. You're asking me so these are *my* opinions - and I do not claim to be anything more than a hobbyist/tinkerer when the comes to this. mbox has potential use for long term read-only archives - I see no reason to use it for live mailboxes. maildir is undoubtedly the least susceptible to corruption. It's also the slowest format for reading. How slow is "slow" depends on your hardware - it may be imperceptible with enough RAM and SSD's - or it may result in user complaints with large mailboxes. dbox is Dovecot's preferred format. I know Timo has put a lot of effort into it. sdbox is similar to maildir in that each mail is a separate file. mdbox significantly reduces the number of files which can make file-based backups faster. Both dbox formats are dependent on their index files. If you've got good hardware, including a proper UPS, I'd recommend dbox (my server is presently using sdbox). With large mailboxes and file-based backups you'll benefit from mdbox. When reliability is the #1 concern above anything else - use maildir. Depending on your use SIS can have significant impact on storage requirements - but storage these days is relatively cheap. I haven't seen much feedback from users actively using SIS - I'd love to hear from high traffic sites with SIS experience to know if the corruption issues have been resolved. In my case there was at least a 30% reduction in space but I had too many errors - admittedly it's been a couple years since I last tried it. -- Daniel
Re: Sis to deduplicate attachments does not work?
On 4/23/2019 1:53 AM, luckydog xf via dovecot wrote: Hi, I use sis to deduplicate attachments, here is my `doveconf -n` [...] mail_location = maildir:/var/mail/%n/Maildir [...] SIS is a function of dbox. You're using Maildir. -- Daniel
Sis to deduplicate attachments does not work?
Hi, I use sis to deduplicate attachments, here is my `doveconf -n`
auth_cache_size = 10 M
auth_failure_delay = 5 secs
auth_mechanisms = plain login
auth_socket_path = /var/run/dovecot/auth-userdb
auth_ssl_username_from_cert = yes
auth_username_format = %Ln
default_vsz_limit = 512 M
lmtp_rcpt_check_quota = yes
mail_attachment_detection_options = add-flags-on-save
mail_attachment_dir = /var/mail/mail_attachment/
mail_debug = yes
mail_gid = vmail
mail_location = maildir:/var/mail/%n/Maildir
mail_plugins = quota fts fts_solr
mail_privileged_group = vmail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate mime foreverypart extracttext
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
fts = solr
fts_autoindex = yes
fts_autoindex_exclude = \Junk
fts_autoindex_exclude2 = \Trash
fts_solr = break-imap-search url=http://localhost:8983/solr/dovecot/
quota = maildir:User quota
quota_grace = 10%%
quota_max_mail_size = 5M
quota_over_flag_lazy_check = yes
quota_over_flag_value = TRUE
quota_rule = *:storage=100M
quota_warning = storage=65%% quota-warning 65 %u
quota_warning2 = storage=60%% quota-warning 60 %u
sieve = file:~/sieve;active=~/.dovecot.sieve
}
protocols = imap lmtp sieve
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0640
user = postfix
}
unix_listener auth-userdb {
mode = 0777
user = $default_internal_user
}
user = $default_internal_user
}
service imap-login {
inet_listener imap {
port = 0
}
inet_listener imaps {
port = 993
ssl = yes
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
process_min_avail = 0
service_count = 1
vsz_limit = 128 M
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 0
}
}
service quota-warning {
executable = script /usr/local/bin/quota-warning.sh
unix_listener quota-warning {
user = vmail
}
user = vmail
}
service stats {
unix_listener stats-writer {
mode = 0666
}
}
service submission-login {
inet_listener submission {
port = 0
}
}
ssl = required
ssl_cert =
