Re: v2.2.16 release candidate released

[Timo Sirainen]

On 09 Mar 2015, at 13:07, Hardy Flor  wrote:
> 
> This easy to correct errors is unfortunately still exists.
> I once hung my solution, but I can only test on a system.
> 
> 
> # dovecot --version
> 2.2.16.rc1
> # doveadm -f table mailbox status "messages guid" -u flor_hardy "*"
> mailbox   messages guid
> Trash 060a619171f66fd5403584b126f5b
> Entwürfe 062a619171f66fd5403584b126f5b
> INBOX 061a619171f66fd5403584b126f5b
> 
> 

Thanks, committed with some coding style changes: 
http://hg.dovecot.org/dovecot-2.2/rev/8780ca0fbf22 

Re: v2.2.16 release candidate released

[Hardy Flor]

This easy to correct errors is unfortunately still exists.
I once hung my solution, but I can only test on a system.


# dovecot --version
2.2.16.rc1
# doveadm -f table mailbox status "messages guid" -u flor_hardy "*"
mailbox   messages guid
Trash 060a619171f66fd5403584b126f5b
Entwürfe 062a619171f66fd5403584b126f5b
INBOX 061a619171f66fd5403584b126f5b

diff -Nubr src.org/doveadm/doveadm-print-table.c src/doveadm/doveadm-print-table.c
--- src.org/doveadm/doveadm-print-table.c	2015-01-29 17:01:15.0 +0100
+++ src/doveadm/doveadm-print-table.c	2015-03-09 11:53:34.041973877 +0100
@@ -46,6 +46,22 @@
 	thdr->flags = hdr->flags;
 }
 
+static size_t utf8len(const char *s)
+{
+  size_t len = 0;
+  while(*s)
+len += (*(s++)&0xC0)!=0x80;
+  return len;
+}
+
+static size_t utf8correction(const char *s)
+{
+  size_t len = 0;
+  while(*s)
+len += (*(s++)&0xC0)==0x80;
+  return len;
+}
+
 static void doveadm_calc_header_length(void)
 {
 	struct doveadm_print_table_header *headers;
@@ -64,7 +80,7 @@
 	for (line = 0; line < line_count; line++) {
 		for (i = 0; i < hdr_count; i++) {
 			value = values[line*hdr_count + i];
-			len = value == NULL ? 0 : strlen(value);
+			len = value == NULL ? 0 : utf8len(value);
 			if (headers[i].min_length > len)
 headers[i].min_length = len;
 			if (headers[i].max_length < len) {
@@ -123,9 +139,9 @@
 	hdr = array_idx(&ctx->headers, ctx->hdr_idx);
 
 	if ((hdr->flags & DOVEADM_PRINT_HEADER_FLAG_RIGHT_JUSTIFY) == 0)
-		printf("%-*s", (int)hdr->length, value);
+		printf("%-*s", (int)(hdr->length+utf8correction(value)), value);
 	else
-		printf("%*s", (int)hdr->length, value);
+		printf("%*s", (int)(hdr->length+utf8correction(value)), value);
 
 	if (++ctx->hdr_idx == array_count(&ctx->headers)) {
 		ctx->hdr_idx = 0;
--- Begin Message ---
In calculating the column width multi-byte characters (eg umlauts, utf8) 
not properly analyzed:


Username mailboxmessages guid
sh_berger Spam   0 a7b32818c14e5b54bf556ad9a53c
sh_berger Entwürfe  0 a4b32818c14e5b54bf556ad9a53c
sh_berger Papierkorb 0 a6b32818c14e5b54bf556ad9a53c
--- End Message ---

Re: v2.2.16 release candidate released

[Hardy Flor]

Instead of deleting it, it should be marked "deleted" by renaming only. 
If it is then required, so it can be recovered. This directory can be 
deleted by another control process that searches the mbox files for 
references.




What has become of the question that attachments are not only saved as 
separate files? (mdbox-files over 2 MB from 17.03.2014)




Am 07.03.2015 um 17:33 schrieb Timo Sirainen:
... When mail is deleted, the hard link is deleted. If the link count 
had been 2, the original attachment file was deleted also.

Re: v2.2.16 release candidate released

[Timo Sirainen]

On 06 Mar 2015, at 21:58, Daniel Miller  wrote:
> 
> On 3/6/2015 7:53 AM, Timo Sirainen wrote:
>> http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz
>> http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz.sig
>> 
>> Looks like it's been a long time since v2.2.15. There have been a ton of 
>> changes since it was released though, so here's a release candidate first to 
>> find out if somebody can find any bugs before the final v2.2.16.
>> 
>> Unfortunately I haven't had time/energy to read Dovecot mailing list for a 
>> while now. I'm hoping this will change, but I don't really expect it to 
>> happen anytime soon. On the positive side for Dovecot, it's now becoming 
>> used in more and more multi-million user installations, which brings all 
>> kinds of nice new improvements.
> 
> Great to hear both Dovecot and you are doing well.  I do need to ask you to 
> check the list for two threads:
> 
> mdbox attachment errors
> Rebuilding SIS attachment links from log
> 
> A few of us have been having SIS problems.

Unless there's a way to reproduce a bug I don't think I can do anything about 
it (I could spend hours looking at the code or trying to reproduce it and come 
up with nothing). But a while ago I did think about a SIS redesign that would 
make it much less likely to break - just need to get it actually implemented:

Currently single instance storage works by having one global directory that 
contains all the attachments. They are hashed by the attachment content, so for 
example /var/attachments/ac/7d/ac7d1274891248912489124 would be the attachment. 
Then each instance would have its own hard link to it, e.g. 
/var/attachments/ac/7d/hashes/ac7d1274891248912489124-1234567890. sdbox and 
mdbox can use these by containing the "ac7d1274891248912489124-1234567890" in 
the header metadata. When mail is deleted, the hard link is deleted. If the 
link count had been 2, the original attachment file was deleted also. (There's 
of course some race conditions here, but in those rare situations the 
attachment would just be duplicated, which isn't too bad.)

The main problem with the old design is that all the users' attachments are 
dumped into a single global directory. It's difficult to take backups and in 
general it seems too difficult to manage correctly so I haven't really 
recommended using it in any bigger installations.

So here's the new idea, which is nearly the same as the old, but with a small 
change that makes it much nicer I think:

Instead of storing the attachment hard links to a global dir, store the hard 
links under the user's mail dir. This way taking backups doesn't require 
anything complicated, just tar the user's mail dir. You can rm -rf the user 
without forever leaving the user's attachments lying around in the global dir 
(assuming there's a job that periodically cleans out attachments with link 
count=1). In general there's no easy way to accidentally break things.

The only new complication here is that if users are split to multiple 
filesystems, hard linking across them isn't going to work. So this would then 
require not only having a per-user mail directory but also per-user attachment 
directory (which would actually be the per-filesystem attachment dir).

The SIS is implemented as lib-fs backend wrapper, so a new one could be 
implemented easily without breaking the old one.

Re: v2.2.16 release candidate released

[Daniel Miller]

On 3/6/2015 7:53 AM, Timo Sirainen wrote:

http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz
http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz.sig

Looks like it's been a long time since v2.2.15. There have been a ton of 
changes since it was released though, so here's a release candidate first to 
find out if somebody can find any bugs before the final v2.2.16.

Unfortunately I haven't had time/energy to read Dovecot mailing list for a 
while now. I'm hoping this will change, but I don't really expect it to happen 
anytime soon. On the positive side for Dovecot, it's now becoming used in more 
and more multi-million user installations, which brings all kinds of nice new 
improvements.


Great to hear both Dovecot and you are doing well.  I do need to ask you 
to check the list for two threads:


mdbox attachment errors
Rebuilding SIS attachment links from log

A few of us have been having SIS problems.

--
Daniel

Re: v2.2.16 release candidate released

[Timo Sirainen]

On 06 Mar 2015, at 18:47, Ralf Hildebrandt  wrote:
> 
> * Timo Sirainen :
> 
>>  + LMTP server: Added lmtp_user_concurrency_limit setting to limit how
>>many LMTP deliveries can be done concurrently for a single user.
> 
> That's broken! 0 doesn't remove the limit :(

Should have been fixed in the rc1 already: 
http://hg.dovecot.org/dovecot-2.2/rev/9c69c197b527

Re: v2.2.16 release candidate released

[Ralf Hildebrandt]

* Timo Sirainen :

>   + LMTP server: Added lmtp_user_concurrency_limit setting to limit how
> many LMTP deliveries can be done concurrently for a single user.

That's broken! 0 doesn't remove the limit :(

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

v2.2.16 release candidate released

[Timo Sirainen]

http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz
http://dovecot.org/releases/2.2/rc/dovecot-2.2.16.rc1.tar.gz.sig

Looks like it's been a long time since v2.2.15. There have been a ton of 
changes since it was released though, so here's a release candidate first to 
find out if somebody can find any bugs before the final v2.2.16.

Unfortunately I haven't had time/energy to read Dovecot mailing list for a 
while now. I'm hoping this will change, but I don't really expect it to happen 
anytime soon. On the positive side for Dovecot, it's now becoming used in more 
and more multi-million user installations, which brings all kinds of nice new 
improvements.

The largest changes since v2.2.15:

* dbox: Resyncing (e.g. doveadm force-resync) no longer deletes
  dovecot.index.cache file. The cache file was rarely the problem
  so this just caused unnecessary slowness.
* Mailbox name limits changed during mailbox creation: Each part of
  a hierarchical name (e.g. "x" or "y" in "x/y") can now be up to 255
  chars long (instead of 200). This also reduces the max number of
  hierarchical levels to 16 (instead of 20) to keep the maximum name
  length 4096 (a common PATH_MAX limit). The 255 char limit is
  hopefully large enough for migrations from all existing systems.
  It's also the limit on many filesystems.

+ director: Added director_consistent_hashing setting to enable
  consistent hashing (instead of the mostly-random MD5 hashing).
  This causes fewer user moves between backends when backend counts
  are changed, which may improve performance (mainly due to caching).
+ director: Added support for "tags", which allows one director ring
  to serve multiple backend clusters with different sets of users.
+ LMTP server: Added lmtp_user_concurrency_limit setting to limit how
  many LMTP deliveries can be done concurrently for a single user.
+ LMTP server: Added support for STARTTLS command.
+ If logging data is generated faster than it can be written, log a
  warning about it and show information about it in log process's
  process title in ps output. Also don't allow a single service to
  flood too long at the cost of delaying other services' logging.
+ stats: Added support for getting global statistics.
+ stats: Use the same session IDs as the rest of Dovecot.
+ stats: Plugins can now create their own statistics fields
+ doveadm server: Non-mail related commands can now also be used
  via doveadm server (TCP socket).
+ doveadm proxying: passdb lookup can now override doveadm_port and
  change the username.
+ doveadm: Search query supports now "oldestonly" parameter to stop
  immediately on the first non-match. This can be used to optimize:
  doveadm expunge mailbox Trash savedbefore 30d oldestonly
+ doveadm: Added "save" command to directly save mails to specified
  mailbox (bypassing Sieve).
+ doveadm fetch: Added body.snippet field, which returns the first
  100 chars of a message without whitespace or HTML tags. The result
  is stored into dovecot.index.cache, so it can be fetched efficiently.
+ dsync: Added -t  parameter to sync only mails newer than
  the given received-timestamp.
+ dsync: Added -F [-] parameter to sync only mails with[out] the
  given flag/keyword.
+ dsync: Added -a  parameter to specify the virtual mailbox
  containing user's all mails. If this mailbox is already found to
  contain the wanted mail (by its GUID), the message is copied from
  there instead of being re-saved. (This isn't efficient enough yet
  for incremental replication.)
+ dsync: -m parameter can now specify \Special-use names for mailboxes.
+ imapc: Added imapc_features=gmail-migration to help migrations from
  GMail. See http://wiki2.dovecot.org/Migration/Gmail
+ imapc: Added imapc_features=search to support IMAP SEARCH command.
  (Currently requires ESEARCH support from remote server.)
+ expire plugin: Added expire_cache=yes setting to cache most of the
  database lookups in dovecot index files.
+ quota: If overquota-flag in userdb doesn't match the current quota
  usage, execute a configured script.
+ redis dict: Added support for expiring keys (:expire_secs=n) and
  specifying the database number (:db=n)
- auth: Don't crash if master user login is attempted without
  any configured master=yes passdbs
- Parsing UTF-8 text for mails could have caused broken results
  sometimes if buffering was split in the middle of a UTF-8 character.
  This affected at least searching messages.
- String sanitization for some