Re: SIS: dovecot creates duplicated attachments (sometimes)
It's been 2 years since my previous post, but the issue is still present (Dovecot 2.3.2). Long story short, SIS attachment deduplication was broken in Dovecot v2.2.16. When it was fixed in v2.2.22 I wrote this script to deduplicate attachments on my servers back: https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck Sometimes I run the script and it finds new duplicated attachments (no too many). If you check these attachments, you find out that the attachments has the same hash, the same size, match if compare them byte by byte, were created at the same time, but one of them has a different inode number. For instance, 3 days ago it was: Attachments processed: 31535 Duplicated attachments: 370 Missed hash files: 1313 And today: Attachments processed: 32368 Duplicated attachments: 381 Missed hash files: 1317 So, 11 new duplicates and 4 new missed hashes. Can someone suggest a way to debug this problem somehow? As a lot of time has passed since my previous post, I repost it below. On 5/16/2016 9:27 AM, Alexander Moisseev wrote: SIS deduplication was broken in 2.2.16 and has been fixed with https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1 but still sometimes (just several times in a month or so) dovecot creates duplicated attachments. As you can see in directory listings below all attachments was created at (about) the same time, but one of them has different inode number. It easy to discover using https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck # dsisck -n mail_uid=vmail mail_attachment_dir=/vmail/attachments ==> Checking SIS... # ln -f ./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd ./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd # mkdir -p -m 700 ./64/8f/hashes && ln ./64/8f/648f5cfa27af6d20c8570fdcaeab997663e15105-55430d2cd4432c571cb63d96bafd ./64/8f/hashes/648f5cfa27af6d20c8570fdcaeab997663e15105 # ln -f ./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd ./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd # ln -f ./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd ./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd -- Unexpected objects found: 0 Attachments processed: 43500 Different attachments with similar hashes skipped: 0 Attachments deduplicated: 3 File system blocks freed up: 1800 Attachment deduplication attempts failed: 0 Hash files created: 1 Hash files creation attempts failed: 0 Hash files re-linked: 0 Hash files re-link attempts failed: 0 Orphaned hash files: 0 Invalid hash files (not a regular file): 0 -- # cd /vmail/attachments # ls -li ./30/b3/ ./30/b3/hashes/ ./30/b3/: total 1996 7464083 -rw--- 1 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-bb6eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-bd6eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-bf6eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-c36eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd 7464084 drwx-- 2 vmail vmail 512 May 10 09:16 hashes ./30/b3/hashes/: total 332 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545 # ls -li ./6f/3f/ ./6f/3f/hashes/ ./6f/3f/: total 644 15088414 -rw--- 1 vmail vmail 326656 May 13 11:23 6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd 15088422 -rw--- 2 vmail vmail 326656 May 13 11:23 6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd 15088448 drwx-- 2 vmail vmail 512 May 13 11:23 hashes ./6f/3f/hashes/: total 320 15088422 -rw--- 2 vmail vmail 326656 May 13 11:23 6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe # ls -li ./dd/d9/ ./dd/d9/hashes/ ./dd/d9/: total 748 80548 -rw--- 3 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-68f7c122fefb3157ad353d96bafd 80548 -rw--- 3 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd 80547 -rw--- 1 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd 80549 drwx-- 2 vmail vmail 512 May 10 18:19 hash
Re: /var/run/dovecot permission issues
On 8/17/2017 7:07 PM, Larry Rosenman wrote: In /usr/local/etc/dovecot/conf.d/90-plugin.conf: It should be enough to just set permissions as other options are defaults. /usr/local/etc/dovecot/conf.d/10-master.conf : service stats { fifo_listener stats-mail { mode = 0666 } fifo_listener stats-user { mode = 0666 } unix_listener stats { mode = 0666 } } BTW I'm not sure if write permissions on 'stats-user' and 'stats' listeners are required for metrics service. At least I have no evidence if Dovecot ever tried to write to that listeners. Probably it is enough to set write permissions on 'stats-mail'.
Re: External mail attachments storage cleanup
On 29.06.2016 13:06, Николай Мананков wrote: Hi! Thanks it worked! But only when I have a specific user instead wldcard (i mean -u *@example.org). Now I think I need to write a script that searches for users in the domain , and starts each of them this command . And apparently it is necessary to add to the cron job.. Hi, The script: doveadm-expunge - iterates over passwd-file databases, expunges messages in Junk and Trash folders and purges mailboxes for every user. https://github.com/moisseev/doveadm-tools/blob/master/bin/doveadm-expunge The crontab entry: 5 4 * * * /usr/local/bin/doveadm-expunge
SIS: dovecot creates duplicated attachments (sometimes)
SIS deduplication was broken in 2.2.16 and has been fixed with https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1 but still sometimes (just several times in a month or so) dovecot creates duplicated attachments. As you can see in directory listings below all attachments was created at (about) the same time, but one of them has different inode number. It easy to discover using https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck # dsisck -n mail_uid=vmail mail_attachment_dir=/vmail/attachments ==> Checking SIS... # ln -f ./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd ./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd # mkdir -p -m 700 ./64/8f/hashes && ln ./64/8f/648f5cfa27af6d20c8570fdcaeab997663e15105-55430d2cd4432c571cb63d96bafd ./64/8f/hashes/648f5cfa27af6d20c8570fdcaeab997663e15105 # ln -f ./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd ./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd # ln -f ./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd ./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd -- Unexpected objects found: 0 Attachments processed: 43500 Different attachments with similar hashes skipped: 0 Attachments deduplicated: 3 File system blocks freed up: 1800 Attachment deduplication attempts failed: 0 Hash files created: 1 Hash files creation attempts failed: 0 Hash files re-linked: 0 Hash files re-link attempts failed: 0 Orphaned hash files: 0 Invalid hash files (not a regular file): 0 -- # cd /vmail/attachments # ls -li ./30/b3/ ./30/b3/hashes/ ./30/b3/: total 1996 7464083 -rw--- 1 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-bb6eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-bd6eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-bf6eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-c36eae04a67c3157287f01003d96bafd 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd 7464084 drwx-- 2 vmail vmail 512 May 10 09:16 hashes ./30/b3/hashes/: total 332 7464082 -rw--- 6 vmail vmail 337719 May 10 09:16 30b367c584a123eee59478adf3e4f4c9e1226545 # ls -li ./6f/3f/ ./6f/3f/hashes/ ./6f/3f/: total 644 15088414 -rw--- 1 vmail vmail 326656 May 13 11:23 6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd 15088422 -rw--- 2 vmail vmail 326656 May 13 11:23 6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd 15088448 drwx-- 2 vmail vmail 512 May 13 11:23 hashes ./6f/3f/hashes/: total 320 15088422 -rw--- 2 vmail vmail 326656 May 13 11:23 6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe # ls -li ./dd/d9/ ./dd/d9/hashes/ ./dd/d9/: total 748 80548 -rw--- 3 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-68f7c122fefb3157ad353d96bafd 80548 -rw--- 3 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd 80547 -rw--- 1 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd 80549 drwx-- 2 vmail vmail 512 May 10 18:19 hashes ./dd/d9/hashes/: total 248 80548 -rw--- 3 vmail vmail 250640 May 10 18:19 ddd97aa6f624d4f54968d2c4956fc3a9d796b31b # doveconf -n # 2.2.24 (a82c823): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.14 (099a97c) # OS: FreeBSD 10.3-RELEASE i386 auth_default_realm = example.com auth_mechanisms = digest-md5 cram-md5 plain apop doveadm_password = # hidden, use -P to show it first_valid_gid = 1000 first_valid_uid = 1000 lda_mailbox_autosubscribe = yes listen = * mail_attachment_dir = /vmail/attachments mail_gid = vmail mail_home = /vmail/%d/%n mail_location = mdbox:~/mdbox mail_plugins = quota zlib acl mail_shared_explicit_inbox = yes mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { list = children location = mdbox:%%h/mdbox:INDEXPVT=~/mdbox/shared/%%u prefix = shared/%%u/ separator
Re: sis deduplication broken from 2.2.16 upwards
On 11.03.2016 3:56, Timo Sirainen wrote: So, after the fix is applied, does dovecot silently delete the duplicated files, or is there a command that needs to be run manually? You'd have to do it manually in some way. A script that does something like: Go through all attachment directories and for each file: - Sort files by filename - Identify that files A and B the same (beginning of the filename begins with same hash), but have a different inode - ln A B.tmp && mv B.tmp B The problem turned out to be a bit more complicated than that. Finally a came up with that script: https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck It assumes Dovecot should not run.
Re: How to delete Dovecot users when using Single Instance Storage
On 19.03.2016 18:56, Jesus Cea wrote: In case somebody find this useful: https://blog.jcea.es/posts/20160211-delete_users_dovecot.html "The expunge step" didn't work because "all" parameter was broken in 2.2.21. It's fixed now: https://github.com/dovecot/core/commit/6971937a6f3e93844dbd43bdbe903628e21a9422 Also as workaround you can replace 'all' with '1:*'. Here is a script that automates the procedure: https://github.com/moisseev/doveadm-tools/blob/master/bin/doveadm-mdboxrm
Re: sis deduplication broken from 2.2.16 upwards
On 11.03.16 3:56, Timo Sirainen wrote: On 11 Mar 2016, at 02:37, Charles Marcuswrote: On 3/9/2016 9:02 PM, Timo Sirainen wrote: On 08 Mar 2016, at 01:50, Pavel Stano wrote: sis attachment deduplication is broken in 2.2.16 upwards. It is caused by this commit. https://github.com/dovecot/core/commit/664bf3e236c214aee86294483c379e4fa66c2e63 in src/lib-fs/fs-sis.c function fs_sis_try_link() is comparation of inodes of hash files. Because fs_stat() after that commit use fstat() on open fd of temporary file instead of stat on filename. But that temporary file has differnt inode. It not cause any corruption but it will not save any space. Because every duplicate attachment will be in separate file. Thanks, fixed: https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1 So, after the fix is applied, does dovecot silently delete the duplicated files, or is there a command that needs to be run manually? You'd have to do it manually in some way. A script that does something like: Go through all attachment directories and for each file: - Sort files by filename - Identify that files A and B the same (beginning of the filename begins with same hash), but have a different inode - ln A B.tmp && mv B.tmp B I've also found that many of /hashes/ directories have missed. # ll /tank1/vmail/attachments/1f/1f total 3300 -rw--- 1 vmail vmail 403976 12 ноя 00:20 1f1f504c582600a2af94b39c088692aba714fe72-c53b9e1508b14356797d0100d09efc50 -rw--- 1 vmail vmail 403976 12 ноя 00:20 1f1f504c582600a2af94b39c088692aba714fe72-c93b9e1508b14356797d0100d09efc50 -rw--- 1 vmail vmail 403976 12 ноя 00:20 1f1f504c582600a2af94b39c088692aba714fe72-f2a777181eb14356807d0100d09efc50 -rw--- 1 vmail vmail 403976 12 ноя 00:20 1f1f504c582600a2af94b39c088692aba714fe72-f31a5e2917b143567e7d0100d09efc50 -rw--- 1 vmail vmail 2582016 3 ноя 00:20 1f1f97880e8cddc2dfe3c4ad2654b9da937226b7-94c53d358bd33756d614d09efc50 Is it related to the same bug or there is another issue? Is it safe to delete attachment files if there is no file with the same hash in the /hashes/ directory or there is no /hashes/ directory at all?
Re: [bug][Pigeonhole 0.4.10] Missing msgid
On 22.12.15 2:06, Stephan Bosch wrote: I am quite confident that this fixes it: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/37ddc42c007c Regards, Stephan. Yes, that change fixed the issue. Thanks!
Re: [bug][Pigeonhole 0.4.10] Missing msgid
On 16.12.15 12:32, Stephan Bosch wrote: That means that LTMP performed that delivery. That would normally only happen when an error occurs. No errors. All mail is delivering as expected. Do you see anything strange in the user log? Does system log show anything more when mail_debug=yes is active? There is nothing unusual in the log. Dec 16 12:43:29 mx dovecot: lmtp(72024): Connect from local Dec 16 12:43:29 mx dovecot: lmtp(use...@example.com): bD6dNzIycVZYGQEAPZa6/Q: sieve: stored mail into mailbox 'INBOX' Dec 16 12:43:29 mx postfix/lmtp[72023]: 7C9D836: to=, relay=mx.mezonplus.ru[private/dovecot-lmtp], delay=0.05, delays=0.02/0/0/0.02, dsn=2.0.0, status=sent (250 2.0.0 bD6dNzIycVZYGQEAPZa6/Q Saved) Dec 16 12:43:29 mx dovecot: lmtp(72024): Disconnect from local: Successful quit Dec 16 12:43:29 mx postfix/qmgr[70108]: 7C9D836: removed What is your configuration? Attached. mx# doveadm config -n # 2.2.21 (5345f22): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.10 (d61ff8a5af9e+) # OS: FreeBSD 10.2-RELEASE i386 auth_default_realm = mezonplus.ru auth_mechanisms = digest-md5 cram-md5 plain apop doveadm_password = # hidden, use -P to show it first_valid_gid = 1000 first_valid_uid = 1000 lda_mailbox_autosubscribe = yes listen = * mail_attachment_dir = /vmail/attachments mail_gid = vmail mail_home = /vmail/%d/%n mail_location = mdbox:~/mdbox mail_plugins = quota zlib acl mail_shared_explicit_inbox = yes mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart namespace { list = children location = mdbox:%%h/mdbox:INDEXPVT=~/mdbox/shared/%%u prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Archives { special_use = \Archive } mailbox Drafts { special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = scheme=plain username_format=%n /usr/local/etc/dovecot/dovecot.auth/%d.passwd driver = passwd-file } plugin { acl = vfile acl_shared_dict = file:/vmail/shared-mailboxes.db antispam_backend = spool2dir antispam_spam = Junk antispam_spool2dir_notspam = /tmp/spamspool/%%020lu-%u-%%05luh antispam_spool2dir_spam = /tmp/spamspool/%%020lu-%u-%%05lus antispam_trash = Trash;train_ham;train_prob;train_spam quota = dict:User quota::file:%h/dovecot-quota quota_rule = *:storage=2G quota_rule2 = Trash:storage=+10%% quota_rule3 = Spam:storage=+20%% quota_status_nouser = DUNNO quota_status_success = DUNNO sieve_after = /usr/local/etc/dovecot/sieve/sieve.after sieve_before = /usr/local/etc/dovecot/sieve/sieve.before sieve_vacation_min_period = 0 zlib_save = gz zlib_save_level = 3 } postmaster_address = postmas...@mezonplus.ru protocols = imap lmtp sieve pop3 quota_full_tempfail = yes service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { user = vmail } } service config { unix_listener config { mode = 0600 user = vmail } } service imap-login { inet_listener imaps { port = 0 } process_limit = 200 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = vmail } service pop3-login { inet_listener pop3s { port = 0 } } service quota-status { client_limit = 1 executable = quota-status -p postfix unix_listener /var/spool/postfix/private/quota-status { user = postfix } } ssl_cert =
[bug][Pigeonhole 0.4.10] Missing msgid
Pigeonhole 0.4.10 doesn't log msgid for some messages. Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): EbLlL2gLcVYCBgEAPZa6/Q: sieve: msgid=<20151216065756.be1df1f...@cp.bitrix24.com>: forwarded toDec 16 09:57:57 mx dovecot: lmtp(us...@example.com): EbLlL2gLcVYCBgEAPZa6/Q: sieve: stored mail into mailbox 'INBOX' Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): FrLlL2gLcVYCBgEAPZa6/Q: sieve: msgid=<20151216065756.c77bb1f...@cp.bitrix24.com>: forwarded to Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): FrLlL2gLcVYCBgEAPZa6/Q: sieve: msgid=<20151216065756.c77bb1f...@cp.bitrix24.com>: stored mail into mailbox 'bitrix'
2.2.21: doveadm expunge is broken
# doveadm expunge -u u...@example.com MAILBOX Trash ALL Fatal: expunge: To avoid accidents, each branch in search query must contain something else besides MAILBOX (e.g. just add "all" if you want everything) -- Alexander
Re: users of dbox format
27.02.2015 18:14, Paolo Cravero пишет: mdbox is THE reason why I am trying Dovecot. With mailboxes of several (tens of) GB with several k of messages I hope mdbox will speedup backups. What backup method are you using? I am doing one-way synchronization to mdbox+SIS at separate ZFS filesystem of the backup server and make snapshots to have backup history. I used to backup with rsync previously. After was I switched to doveadm-backup the backup speed increased dramatically.
Re: users of dbox format
27.02.2015 17:42, Andreas Kasenides пишет: I am interested in finding out your experiences with using the dbox format (especially mdbox) if you use this format. I am using mdbox+SIS on small production server about 2 months. It works well for me. But I have had some difficulties using doveadm tools in particular situations. Mostly because of some parts of documentation wasn't clear enough for me. I am contemplating changing my maildir setup to mdbox but I still need to make a case for it against maildir which has become a de-facto standard and provides sort of a secure basis in case of software changes. You can convert from mdbox to maildir at any time. Also you can use both formats on the same server simultaneously. -- Alexander
Re: curious when certain patches might become part of a release
13.02.2015 20:24, Benjamin Connelly пишет: We ran in to this same problem others discussed in this thread: http://dovecot.org/list/dovecot/2014-November/098927.html and have also applied the patches (31262a892ba7 and 80ed82a93c1a) and it seems to have stopped the panics. We usually just use the FreeBSD ports tree to install software, so I'm looking forward to the time when we can go back to that. Will those patches be part of the next release? you have to ask this question the FreeBSD maintainer of the Dovecot package. But the FreeBSD ports tree is up to the latest 2.2.15 -- yet we were still experiencing the panics, and had to apply those patches. So I'm wondering more about what version release of Dovecot itself will include those mbox sync patches. . . Ben Do you mind to open a PR in the FreeBSD Bugzilla? -- Alexander
Re: curious when certain patches might become part of a release
13.02.2015 21:38, Benjamin Connelly пишет: The bug wasn't specific to FreeBSD (or the FreeBSD port.) It's apparently fixed by 31262a892ba7 and 80ed82a93c1a Of course it is not a port bug. I minded you can ask port maintainer to include those patches into the port.
Re: How to Delete an user and Purge attachements with a common directory mail_attachment_dir with SIS for all users.
On 12.02.2015 12:00, TN wrote: Hello Alexander, Thank you for your answer, it is very helpfull for me. Just a little addon with your process to complet the deletion an user : # doveadm expunge -d -u johndoe mailbox '*' all # doveadm -v purge -u johndoe (this command force attachment deletion immediatly ) # rm -rf /var/mail/johndoe Thank you PS: Sorry to not reply to the dovecot list because i setup a digest for all answers. Hi, Many thanks for your reply. It's a *very important* addition actually. There is lack of documentation on SIS. So I have done some experimenting and found out that the command # doveadm purge -u johndoe *is mandatory*. Without this step attachements will stay in mail_attachment_dir forever. Man doveadm-purge(1) states doveadm-purge - Remove messages with refcount=0 from mdbox files, but actually it also removes attachments from mail_attachment_dir. -- Alexander
Re: How to Delete an user and Purge attachements with a common directory mail_attachment_dir with SIS for all users.
10.02.2015 18:36, TN пишет: What is the best way to delete an user mailbox (ex: /var/mail/johndoe ) when all attachments (for all users) are in a common directory with SIS deduplication (ex: mail_attachment_dir = /var/mail/attachments ) ? Trying to delete user mailbox directory (rm) and do the command : doveadm -v purge -u johndoeleave all johndoe's attachments orphelin. You should always expunge mailbox *before* deleting it, if you are using SIS. # doveadm expunge -d -u johndoe mailbox '*' all # rm -rf /var/mail/johndoe https://www.mail-archive.com/dovecot@dovecot.org/msg41683.html -- Alexander
[Bug ?] IMAP keywords are not preserved during mailbox conversion
IMAP keywords (Thunderbird Tags) are not preserved during Maildir to mdbox conversion. For mailbox conversion I am using the command: # doveadm -o mail_location=maildir:~/Maildir sync -u u...@example.com mdbox:~/mdbox After conversion all Thunderbird Tags are lost. Is it a bug or expected behavior? -- Alexander # doveconf -n # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 (3e924b1b6c5c+) # OS: FreeBSD 10.1-RELEASE i386 auth_default_realm = example.com auth_mechanisms = digest-md5 cram-md5 apop plain doveadm_password = first_valid_gid = 1000 first_valid_uid = 1000 lda_mailbox_autosubscribe = yes listen = * mail_attachment_dir = /vmail/attachments mail_gid = vmail mail_home = /vmail/%d/%n mail_location = mdbox:~/mdbox mail_plugins = quota zlib mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = scheme=plain username_format=%n /usr/local/etc/dovecot/dovecot.auth/%d.passwd driver = passwd-file } plugin { antispam_backend = mailtrain antispam_mail_notspam = learn_ham antispam_mail_sendmail = /usr/local/bin/rspamc antispam_mail_sendmail_args = -h;localhost:11334 antispam_mail_spam = learn_spam antispam_spam = Junk antispam_trash = Trash quota = dict:User quota::file:%h/dovecot-quota quota_rule = *:storage=2G quota_rule2 = Trash:storage=+10%% quota_rule3 = Spam:storage=+20%% quota_status_nouser = DUNNO quota_status_success = DUNNO sieve_after = /usr/local/etc/dovecot/sieve/sieve.after sieve_before = /usr/local/etc/dovecot/sieve/sieve.before sieve_vacation_min_period = 0 zlib_save = gz zlib_save_level = 3 } postmaster_address = postmas...@example.com protocols = imap pop3 lmtp sieve quota_full_tempfail = yes service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { user = vmail } } service config { unix_listener config { mode = 0600 user = vmail } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = vmail } service quota-status { client_limit = 1 executable = quota-status -p postfix unix_listener /var/spool/postfix/private/quota-status { user = postfix } } ssl_cert = /etc/ssl/certs/mx.example.com.crt ssl_key = /etc/ssl/private/mx.example.com.key userdb { args = username_format=%n /usr/local/etc/dovecot/dovecot.auth/%d.passwd driver = passwd-file } verbose_proctitle = yes protocol imap { mail_plugins = quota zlib antispam imap_quota imap_zlib } protocol lmtp { mail_plugins = quota zlib sieve } #
dsync-remote: Error: Failed to set attribute
1. I am looking for backup solution for mdbox with SIS and experimenting with dsync. I am planning to do it like that: doveadm backup [deSISing] = transmitting to remote host = dsync-server [SISing] Is there more convenient method? 2. I have dovecot 2.2.15 on both hosts. The local host is a mail server. Dovecot on the remote host neither running nor configured (I have created an empty dovecot.conf in the config dir). When I tried to back up mailbox from local to remote host with command doveadm backup -P -u u...@domain.tld \ ssh -c blowfish -i /tmp/vmail/.ssh/id_dsa vmail@remotehost \ doveadm \ -o mail_location=mdbox:/tmp/vmail/domain.tld/user/mdbox \ -o mail_attachment_dir=/tmp/vmail/attachments \ dsync-server I got error messages: dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/files/roundcube: Internal attributes cannot be changed directly dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/files/main: Internal attributes cannot be changed directly dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/files/test: Internal attributes cannot be changed directly dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/default: Internal attributes cannot be changed directly What do that errors mean? Does dsync try to replicate sieve scripts? -- Alexander
Re: [Dovecot] May 05 07:20:21 imap: Warning: Time jumped forwards 16 seconds
Are you got this warnings only after OS reboot? -- Alexander Moisseev
Re: [Dovecot] Time just moved backwards error even with ntpd
On some FreeBSD 6.2 systems I was solved this problem just by adding to crontab: @hourly/sbin/sysctl `/sbin/sysctl -e machdep.adjkerntz` /dev/null 21 For some explanation, see: http://lists.freebsd.org/pipermail/freebsd-hackers/2005-August/013383.html Quote follows: ... It seems that CMOS clock is updated using resettodr(9) function. There seem to be only a few occasions when this function is called: 1. clock_settime(ClOCK_REALTIME) // through kern_time.c:settime() 2. settimeofday() // through kern_time.c:settime() 3. machdep.adjkerntz sysctl is set I believe that ntpd calls *settime-family* functions *only* if time offset is *larger* than *128ms* (or maybe I am thinking about ntpdate), but normally it uses ntp_adjtime() to adjust time keeping. Obviosuly, a system running ntpd with good enough hardware clock and good enough connection to good enough ntp server(s) would use the latter method all of the time after some initial stabilization period. But it seems that time changes are *never propagated to CMOS* in this case, ... Alexander
[Dovecot] Maildir POP3 UID larger than next_uid bug
dovecot 1.0.0; Free BSD 6.2; x86; UFS; no NFS IMAP not uses at all, POP3 only Problem description: POP3 client can't fetch second e-mail message from Maildir if first POP3 connect occurs before any message placed in Maildir. Details: 1. POP3-client first time tries to receive mail: May 14 16:17:36 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], method=PLAIN, rip=192.168.0.100, lip=192.168.0.3 May 14 16:17:36 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0 dovecot creates Maildir for that user: /var/mail/virtual/mydomain.ru/user/Maildir/ /cur /new /tmp dovecot.index dovecot.index.cache dovecot.index.log 2. Postfix MTA receives message for that user and places it at /var/mail/virtual/mydomain.ru/user/Maildir/new/1179145269.V43I3f5a4M116738.host.mydomain.local 3. POP3-client tries to receive mail: May 14 16:23:09 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], method=PLAIN, rip=192.168.0.100, lip=192.168.0.3 May 14 16:23:09 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out top=0/0, retr=1/786, del=1/1, size=770 dovecot creates file dovecot-uidlist with next_uid=1 (!!!): 1 1179145056 1 1 1179145269.V43I3f5a4M116738.host.mydomain.local:2, Message fetched successfully. 4. POP3-client tries to receive mail: May 14 16:23:29 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], method=PLAIN, rip=192.168.0.100, lip=192.168.0.3 May 14 16:23:29 host dovecot: POP3([EMAIL PROTECTED]): UID larger than next_uid in file /var/mail/virtual/mydomain.ru/user/Maildir/dovecot-uidlist (1 = 1) May 14 16:23:29 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0 File dovecot-uidlist disappears. 5. Postfix MTA receives message for that user and places it at /var/mail/virtual/mydomain.ru/user/Maildir/new/1179145534.V43I3f5a4M116827.host.mydomain.local 6. POP3-client tries to receive mail: May 14 16:47:20 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], method=PLAIN, rip=192.168.0.100, lip=192.168.0.3 May 14 16:47:20 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0 Dovecot creates file dovecot-uidlist: 1 1179145056 2 1 1179145534.V43I3f5a4M116827.host.mydomain.local Dovecot moves message from new to cur directory and adds flag :2,: /var/mail/virtual/mydomain.ru/user/Maildir/cur/1179145534.V43I3f5a4M116827.host.mydomain.local:2, Message not fetched. And POP3 client can't fetch it at all. All next messages fetches normally as they arrives. # /usr/local/etc/dovecot.conf protocols: pop3 pop3s ssl_cert_file: /etc/ssl/certs/host.mydomain.ru.crt ssl_key_file: /etc/ssl/private/host.mydomain.ru.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/pop3-login login_greeting: POP3 server ready verbose_proctitle: yes first_valid_gid: 0 mail_extra_groups: mail mail_location: maildir:~/Maildir mail_executable: /usr/local/libexec/dovecot/pop3 mail_plugin_dir: /usr/local/lib/dovecot/pop3 pop3_uidl_format: %08Xu%08Xv pop3_client_workarounds: outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain login digest-md5 cram-md5 apop ntlm default_realm: mydomain.ru passdb: driver: pam passdb: driver: passwd-file args: /etc/auth/%d/passwd userdb: driver: passwd userdb: driver: static args: uid=5000 gid=5000 home=/var/mail/virtual/%d/%n socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: