Re: v2.2.32 release candidate released
On 18 Aug 2017, at 2.25, Joseph Tam wrote: > >> a CREATE noselectbox/ >> --> With this change it's the same as CREATE "noselectbox" = will become >> selectable. >> >> or >> >> a CREATE noselectbox/childbox >> b DELETE noselectbox/childbox >> --> noselectbox is left behind. With this change it will be auto-deleted. > > So, does that mean empty folders will be auto-deleted, or this wouldn't > affect them? Selectable folders won't be auto-deleted, only \NoSelect folders that have no child folders. This auto-deletion is also done if LIST notices them. Actually the main reason it was implemented now was because with NFS if a folder is open while it's deleted, it could leave .nfs* files in the directory and prevent finishing the deletion. So a folder deletion would change it into a \NoSelect folder, but not delete it entirely. This is mainly a user-visible problem if the new ITERINDEX feature is used.
Re: v2.2.32 release candidate released
Timo Sirainen wrote: Thanks for the reply -- I think I can now write a new config to test these tasty features out. + mail_location can now include NO-NOSELECT parameter. This automatically deletes any \NoSelect mailboxes that have no children. These mailboxes are sometimes confusing to users. Sorry for my IMAP ignorance, but how can this situation come about? a CREATE noselectbox/ --> With this change it's the same as CREATE "noselectbox" = will become selectable. or a CREATE noselectbox/childbox b DELETE noselectbox/childbox --> noselectbox is left behind. With this change it will be auto-deleted. So, does that mean empty folders will be auto-deleted, or this wouldn't affect them? I think already even with mailbox_list_index=yes it won't notice external changes done to the mailboxes when doing STATUS calls, so better to avoid that. It does get fixed automatically once the folder is selected though. I'm running it this way now (concurrent file and dovecot access) and most users are blithe to it, so I guess it won't be too much worse. I'll try aggressive caching and back off if it causes confusion. Thanks. Joseph Tam
Re: v2.2.32 release candidate released
On 16 Aug 2017, at 23.59, Joseph Tam wrote: > > > Timo Sirainen wrote: > >> There are various changes in this release that can be used to significantly >> reduce disk IO with: >> 1) NFS storage especially, but I guess also other remote filesystems and >> even some with local disks >> 2) When mail storage and INDEX storage are separated > > Thanks for these changes! Big win for my setup. My servers are not > overly stressed, but how much performance gain (using any metric you > choose) can one expect from these changes? Depends a lot of things, but for the one specific NFS installation we reduced NFS IOPS by something like 70% And actually they haven't finished setting up the local LISTINDEX changes yet, so maybe even more. >> + mail_location can now include VOLATILEDIR= parameter. This >> is used for creating lock files and in future potentially other >> files that don't need to exist permanently. The path could point to >> tmpfs for example. This is especially useful to avoid creating lock >> files to NFS or other remote filesystems. For example: >> mail_location=sdbox:~/sdbox:VOLATILEDIR=/tmp/volatile/%2.256Nu/%u > > Is "/tmp/volatile" auto-created, or must be pre-created? Autocreated. >> + mail_location's LISTINDEX= can now contain a full path. >> This allows storing mailbox list index to a different storage >> than the rest of the indexes, for example to tmpfs. > > Is this in any way related to VOLATILEDIR? Can they be set to the same > value without problems? Yeah, can be set to the same directory. Maybe even a good idea. >> + mail_location can now include NO-NOSELECT parameter. This >> automatically deletes any \NoSelect mailboxes that have no children. >> These mailboxes are sometimes confusing to users. > > Sorry for my IMAP ignorance, but how can this situation come about? a CREATE noselectbox/ --> With this change it's the same as CREATE "noselectbox" = will become selectable. or a CREATE noselectbox/childbox b DELETE noselectbox/childbox --> noselectbox is left behind. With this change it will be auto-deleted. >> + mail_location can now include ITERINDEX parameter. This tells Dovecot >> to perform mailbox listing from the INDEX path instead of from the >> mail root path. It's mainly useful when the INDEX storage is on a >> faster storage. >> + If mailbox_list_index_very_dirty_syncs=yes, the list index is no >> longer refreshed against filesystem when listing mailboxes. This >> allows the mailbox listing to be done entirely by only reading the >> mailbox list index. >> + Added mailbox_list_index_include_inbox setting to control whether >> INBOX's STATUS information should be cached in the mailbox list >> index. The default is "no", but it may be useful to change it to >> "yes", especially if LISTINDEX points to tmpfs. > > So as I understand it, the optimzation comes about from segregating mail > data information into 3 parts: raw mail, indices, and volatile components, > putting them into increasingly better performing storage media. > > How do these I/O optimizations affect the client's view of a mailbox > if their mailbox is subject to modification outside the dovecot system > (e.g. procmail, mail readers directly modifies mailbox files)? Is there > a trade-off between metadata consistency and performance, or it's a > win-win all around? I think already even with mailbox_list_index=yes it won't notice external changes done to the mailboxes when doing STATUS calls, so better to avoid that. It does get fixed automatically once the folder is selected though.
Re: v2.2.32 release candidate released
On Tue, Aug 15, 2017 at 11:49:29PM +0300, Timo Sirainen wrote: > https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz > https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz.sig In case you are interested (and have the spare cycles), I get the following mostly new warnings with gcc-7 -- Eray /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1082:29: warning: this statement may fall through [-Wimplicit-fallthrough=] dctxPtr->dStage = dstage_storeHeader; ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1085:9: note: here case dstage_storeHeader: ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1194:33: warning: this statement may fall through [-Wimplicit-fallthrough=] dctxPtr->dStage = dstage_decodeCBlock; ^ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/lib/lz4frame.c:1198:9: note: here case dstage_decodeCBlock: ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:453:36: warning: this statement may fall through [-Wimplicit-fallthrough=] case 'r': recursive=1; /* without break */ ~^~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:456:17: note: here case 'm': multiple_inputs=1; ^~~~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:453:36: warning: this statement may fall through [-Wimplicit-fallthrough=] case 'r': recursive=1; /* without break */ ~^~ /var/tmp/portage/app-arch/lz4-1.7.5-r1/work/lz4-1.7.5/programs/lz4cli.c:456:17: note: here case 'm': multiple_inputs=1; ^~~~ json-parser.c:537:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if (parser->skipping && parser->nested_skip_count == 0) { ^ json-parser.c:541:2: note: here case JSON_STATE_ARRAY_NEXT_SKIP: ^~~~ istream-attachment-extractor.c:328:3: warning: this statement may fall through [-Wimplicit-fallthrough=] buffer_set_used_size(part_buf, 0); ^ istream-attachment-extractor.c:330:2: note: here case MAIL_ATTACHMENT_STATE_YES: ^~~~ message-header-encode.c:21:4: warning: this statement may fall through [-Wimplicit-fallthrough=] i++; ~^~ message-header-encode.c:23:2: note: here case '\n': ^~~~ ldap-connection.c:484:16: warning: this statement may fall through [-Wimplicit-fallthrough=] conn->state = LDAP_STATE_AUTH; ^ ldap-connection.c:487:2: note: here case LDAP_STATE_AUTH: ^~~~ fts-filter-contractions.c:46:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if (token[pos] == '\0' || token[pos] != 'u') ^ fts-filter-contractions.c:49:2: note: here case 'c': ^~~~ mail-transaction-log-file.c:1064:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if ((hdr->type & MAIL_TRANSACTION_EXTERNAL) == 0) { ^ mail-transaction-log-file.c:1068:2: note: here case MAIL_TRANSACTION_APPEND: ^~~~ test-mail-transaction-log-file.c:412:18: warning: passing argument 1 of ‘test_run’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers] return test_run(test_functions); ^~ In file included from test-mail-transaction-log-file.c:5:0: ../../src/lib-test/test-common.h:55:5: note: expected ‘void (**)(void)’ but argument is of type ‘void (* const*)(void)’ int test_run(void (*test_functions[])(void)); ^~~~ maildir-save.c:277:30: warning: ‘*’ in boolean context, suggest ‘&&’ instead [-Wint-in-bool-context] mf->keywords_count * sizeof(unsigned int)); ../../../../src/lib/macros.h:161:26: note: in definition of macro ‘COMPILE_ERROR_IF_TRUE’ (sizeof(char[1 - 2 * !!(condition)]) - 1) ^ maildir-save.c:276:2: note: in expansion of macro ‘buffer_create_from_const_data’ buffer_create_from_const_data(&ctx->keywords_buffer, mf + 1, ^ imapc-search.c:99:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if ((mbox->capabilities & IMAPC_CAPABILITY_WITHIN) == 0) { ^ imapc-search.c:110:2: note: here case SEARCH_ALL: ^~~~ mail-search.c:78:7: warning: this statement may fall through [-Wimplicit-fallthrough=] if (arg->value.str == NULL) ^ mail-search.c:81:3: note: here case SEARCH_KEYWORDS: ^~~~ password-scheme.c:186:6: warning: this statement may fall through [-Wimplicit-fallthrough=] if (!guessed_encoding) { ^ password-scheme.c:193:2: note: here case PW_ENCODING_BASE64: ^~~~ client-common-auth.c:58:43: warning: ?: using integer constants in boolean context [-Wint-in-bool-context] AUTH_WA
Re: v2.2.32 release candidate released
Timo Sirainen wrote: There are various changes in this release that can be used to significantly reduce disk IO with: 1) NFS storage especially, but I guess also other remote filesystems and even some with local disks 2) When mail storage and INDEX storage are separated Thanks for these changes! Big win for my setup. My servers are not overly stressed, but how much performance gain (using any metric you choose) can one expect from these changes? + mail_location can now include VOLATILEDIR= parameter. This is used for creating lock files and in future potentially other files that don't need to exist permanently. The path could point to tmpfs for example. This is especially useful to avoid creating lock files to NFS or other remote filesystems. For example: mail_location=sdbox:~/sdbox:VOLATILEDIR=/tmp/volatile/%2.256Nu/%u Is "/tmp/volatile" auto-created, or must be pre-created? + mail_location's LISTINDEX= can now contain a full path. This allows storing mailbox list index to a different storage than the rest of the indexes, for example to tmpfs. Is this in any way related to VOLATILEDIR? Can they be set to the same value without problems? + mail_location can now include NO-NOSELECT parameter. This automatically deletes any \NoSelect mailboxes that have no children. These mailboxes are sometimes confusing to users. Sorry for my IMAP ignorance, but how can this situation come about? + mail_location can now include ITERINDEX parameter. This tells Dovecot to perform mailbox listing from the INDEX path instead of from the mail root path. It's mainly useful when the INDEX storage is on a faster storage. + If mailbox_list_index_very_dirty_syncs=yes, the list index is no longer refreshed against filesystem when listing mailboxes. This allows the mailbox listing to be done entirely by only reading the mailbox list index. + Added mailbox_list_index_include_inbox setting to control whether INBOX's STATUS information should be cached in the mailbox list index. The default is "no", but it may be useful to change it to "yes", especially if LISTINDEX points to tmpfs. So as I understand it, the optimzation comes about from segregating mail data information into 3 parts: raw mail, indices, and volatile components, putting them into increasingly better performing storage media. How do these I/O optimizations affect the client's view of a mailbox if their mailbox is subject to modification outside the dovecot system (e.g. procmail, mail readers directly modifies mailbox files)? Is there a trade-off between metadata consistency and performance, or it's a win-win all around? (I just saw your previous response to someone else, which I'll read more closely.) Joseph Tam
Re: v2.2.32 release candidate released (on Debian 8/Jessie)
Am 15.08.2017 um 22:49 schrieb Timo Sirainen: > https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz my buildsystem complain on spelling errors - in binaries -> src/director/director-connection.c: reseting -> resetting - in manpages -> doc/man/doveadm-exec.1.in: wich -> which + No newline at end of file patches attached. one other point I notice (not new in this versions) are "warnings" from Debian lintian: I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/auth N: N:This package provides an ELF binary that lacks the use of fortified libc N:functions. Either there are no potentially unfortified functions called N:by any routines, all unfortified calls have already been fully validated N:at compile-time, or the package was not built with the default Debian N:compiler flags defined by dpkg-buildflags. If built using N:dpkg-buildflags directly, be sure to import CPPFLAGS. N: N:NB: Due to false-positives, Lintian ignores some unprotected functions N:(e.g. memcpy). N: N:Refer to https://wiki.debian.org/Hardening and N:http://bugs.debian.org/673112 for details. N: N:Severity: normal, Certainty: wild-guess N: N:Check: binaries, Type: binary, udeb N: I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/config I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/director I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/gdbhelper I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/imap I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/libdovecot-login.so.0.0.0 I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib10_quota_plugin.so I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib20_fts_plugin.so I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib20_replication_plugin.so I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/modules/lib99_welcome_plugin.so I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/quota-status I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/script I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/script-login I: dv-dovecot: hardening-no-fortify-functions usr/lib/dovecot/xml2text the text mention "CPPFLAGS" which occour exact two times in my Buildlog: 1. as parameter to configure 2. in that warning I read this as "configure get an evironment variable "CPPFLAGS" and completely ignore them. full buildlog (for Debian Jessie, using aclocal-1.14, btw...) at https://andreasschulze.de/tmp/dv-dovecot_2.2.32~rc1-2017081601_amd64.build.txt Andreas Description: lintian: wich -> which Author: A. Schulze --- This patch header follows DEP-3: http://dep.debian.net/deps/dep3/ Index: dovecot-2.2.32~rc1/doc/man/doveadm-exec.1.in === --- dovecot-2.2.32~rc1.orig/doc/man/doveadm-exec.1.in +++ dovecot-2.2.32~rc1/doc/man/doveadm-exec.1.in @@ -28,7 +28,7 @@ the name of an executable located in .\"- .TP .I binary arguments -options and arguments, wich will be passed through to the +options and arguments, which will be passed through to the .IR binary . .\" .SH EXAMPLE @@ -44,4 +44,4 @@ user\(aqs mailbox. .\" .SH SEE ALSO .BR doveadm (1), -.BR dovecot\-lda (1) \ No newline at end of file +.BR dovecot\-lda (1) Description: lintian: reseting -> resetting Author: A. Schulze --- This patch header follows DEP-3: http://dep.debian.net/deps/dep3/ Index: dovecot-2.2.32~rc1/src/director/director-connection.c === --- dovecot-2.2.32~rc1.orig/src/director/director-connection.c +++ dovecot-2.2.32~rc1/src/director/director-connection.c @@ -770,7 +770,7 @@ static bool director_cmd_director(struct } /* just forward this to the entire ring until it reaches back to itself. some hosts may see this twice, but that's the only way to - guarantee that it gets seen by everyone. reseting the host multiple + guarantee that it gets seen by everyone. resetting the host multiple times may cause us to handle its commands multiple times, but the commands can handle that. however, we need to also handle a situation where the added director never comes back - we don't want @@ -1469,7 +1469,7 @@ director_connection_sync_host(struct dir timestamp); return FALSE; } else if (seq < host->last_sync_seq) { - i_warning("Last SYNC seq for %s appears to be stale, reseting " + i_warning("Last SYNC seq for %s appears to be stale, resetting " "(seq=%u, timestamp=%u -> seq=%u, timestamp=%u)", host->name, host->last_sync_seq, host->last_sync_timestamp, seq, timestamp);
Re: v2.2.32 release candidate released
Hello Timo, Many thanks for the explanation, it really clarified things. I'll go for option (b) "Use mailbox_list_index_very_dirty_syncs=no and put LISTINDEX to local" then and write later on if there are any issues with it.
Re: v2.2.32 release candidate released
On 16 Aug 2017, at 8.37, Alexey Asemov (Alex/AT) wrote: > > Hello Timo, > > I am quite eager to test it, but I don't know if these changes can be > applicable for my configuration (it is shared storage one, any mdbox on NFS > can be accessed by multiple servers, I made it so it's mostly accessed by > one, but at certain conditions multiple servers still access mailbox > simultaneously). > > So if possible I want to ask few dumb questions: > > - I assume VOLATILEDIR can't be used with configurations where multiple > servers access the same mdbox because lockfiles need to be shared as well? Or > does this option touch only some (lock)files that do not need to be shared > between servers accessing same mailbox? If you're using e.g. Dovecot director, it should be very unlikely that multiple servers access the same user simultaneously. Currently only two lock files use VOLATILEDIR: * .vsize.lock: Used to update folder's size in dovecot.index. So the worst that can happen is that the vsize becomes wrong. If you're using quota=count, it could become wrong. But this gets fixed automatically anyway later on. * autoexpunge.lock : Only used if you have autoexpunge settings. > - Would LISTINDEX indexes be regenerated properly for mdbox if > lost/corrupted? Properly, meaning without losing flags, message numbering or > some other vital data. If yes, then I assume they can be stored on local > storage per server, and not on NFS. If LISTINDEX can be regenerated, will > LISTINDEX be updated if it's detected to be obsolete compared to accessed > mdbox contents? dovecot.list.index* don't have any especially vital information, like message flags. So it can be fully regenerated. You have 3 options actually: a) Use mailbox_list_index_very_dirty_syncs=yes and keep LISTINDEX in NFS: This optimizes LIST b) Use mailbox_list_index_very_dirty_syncs=no and put LISTINDEX to local: This automatically updates the list index as necessary, but it does some extra stat()s on the folders to make sure they're up-to-date in the list index. c) Use mailbox_list_index_very_dirty_syncs=yes and put LISTINDEX to local: This fully trusts the locally cached list indexes. It works only if you can guarantee that the local list indexes aren't obsolete. So it requires some scripting. For example you can store in NFS or in Redis or elsewhere the user's last backend hostname. Then create a post-login script that deletes the list indexes if the hostname doesn't match the current server where user is logging in. This can also be optimized to run the script only when the hostname changes by using the new userdb postlogin socket and %{if...} that is coming to v2.2.33. Although there are also other ways.
Re: v2.2.32 release candidate released
Hello Timo, I am quite eager to test it, but I don't know if these changes can be applicable for my configuration (it is shared storage one, any mdbox on NFS can be accessed by multiple servers, I made it so it's mostly accessed by one, but at certain conditions multiple servers still access mailbox simultaneously). So if possible I want to ask few dumb questions: - I assume VOLATILEDIR can't be used with configurations where multiple servers access the same mdbox because lockfiles need to be shared as well? Or does this option touch only some (lock)files that do not need to be shared between servers accessing same mailbox? - Would LISTINDEX indexes be regenerated properly for mdbox if lost/corrupted? Properly, meaning without losing flags, message numbering or some other vital data. If yes, then I assume they can be stored on local storage per server, and not on NFS. If LISTINDEX can be regenerated, will LISTINDEX be updated if it's detected to be obsolete compared to accessed mdbox contents? Thanks. On 15.08.2017 23:49, Timo Sirainen wrote: https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz https://dovecot.org/releases/2.2/rc/dovecot-2.2.32.rc1.tar.gz.sig There are various changes in this release that can be used to significantly reduce disk IO with: 1) NFS storage especially, but I guess also other remote filesystems and even some with local disks 2) When mail storage and INDEX storage are separated