Re: 2.3.1 Replication is throwing scary errors

[Thore Bödecker]

And I forgot to CC the list, sorry for that, it's way too early in
the morning :P

On 07.06.18 - 07:39, Thore Bödecker wrote:
> What does the output of these two commands show after that error has
> been logged?
> 
> doveadm replicator status
> 
> doveadm replicator dsync-status
> 
> If there are *waiting failed* requests, that never make it "through"
> (after being temporarily in state *queued failed* and then returning
> to *waiting failed*) this means there is something wrong with the
> replication.
> 
> You can try forcing replication of all known users using
> 
> doveadm replicator replicate '*'
> 
> And see if that resolves the failed requests, but I doubt it.
> 
> Please let us know how your status outputs look like.
> 
> 
> Cheers,
> Thore
> 
> -- 
> Thore Bödecker
> 
> GPG ID: 0xD622431AF8DB80F3
> GPG FP: 0F96 559D 3556 24FC 2226  A864 D622 431A F8DB 80F3
> 
> 
> On 07.06.18 - 15:21, Reuben Farrelly wrote:
> > Still not quite right for me.
> > 
> > Jun  7 15:11:33 thunderstorm.reub.net dovecot: doveadm: Error:
> > dsync(lightning.reub.net): I/O has stalled, no activity for 600 seconds
> > (last sent=mail, last recv=mail (EOL))
> > Jun  7 15:11:33 thunderstorm.reub.net dovecot: doveadm: Error: Timeout
> > during state=sync_mails (send=mails recv=recv_last_common)
> > 
> > I'm not sure if there is an underlying replication error or if the message
> > is just cosmetic, though.
> > 
> > Reuben
> > 
> > 
> > On 7/06/2018 4:55 AM, Remko Lodder wrote:
> > > Hi Timo,
> > > 
> > > Yes this seems to work fine so far. I’ll ask the people to add it to the
> > > current FreeBSD version..
> > > 
> > > Cheers
> > > Remko
> > > 
> > > > On 6 Jun 2018, at 19:34, Timo Sirainen  > > > > wrote:
> > > > 
> > > > Should be fixed by 
> > > > https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336
> > > > 
> > > 


signature.asc
Description: PGP signature

Re: 2.3.1 Replication is throwing scary errors

[Remko Lodder]

> On 7 Jun 2018, at 07:21, Reuben Farrelly  wrote:
> 
> Still not quite right for me.
> 
> Jun  7 15:11:33 thunderstorm.reub.net dovecot: doveadm: Error: 
> dsync(lightning.reub.net): I/O has stalled, no activity for 600 seconds (last 
> sent=mail, last recv=mail (EOL))
> Jun  7 15:11:33 thunderstorm.reub.net dovecot: doveadm: Error: Timeout during 
> state=sync_mails (send=mails recv=recv_last_common)
> 
> I'm not sure if there is an underlying replication error or if the message is 
> just cosmetic, though.
> 
> Reuben

Hi,

Admittedly I have had a few occurences of this behaviour as well last night. It 
happens more sporadic now and seems to be a conflict with my user settings. (My 
users
get added twice to the system, user-domain.tld and u...@domain.tld 
, both are being replicated, the noreplicate flag is 
not yet honored in the version I am using so I cannot
bypass that yet).

I do see messages that came on the other machine on the machine that I am using 
to read these emails. So replication seems to work in that regard (where it 
obviously
did not do that well before).

Cheers
Remko

> 
> 
> On 7/06/2018 4:55 AM, Remko Lodder wrote:
>> Hi Timo,
>> Yes this seems to work fine so far. I’ll ask the people to add it to the 
>> current FreeBSD version..
>> Cheers
>> Remko
>>> On 6 Jun 2018, at 19:34, Timo Sirainen mailto:t...@iki.fi>> 
>>> wrote:
>>> 
>>> Should be fixed by 
>>> https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336
>>> 



signature.asc
Description: Message signed with OpenPGP

Re: 2.3.1 Replication is throwing scary errors

[Reuben Farrelly]

Still not quite right for me.

Jun  7 15:11:33 thunderstorm.reub.net dovecot: doveadm: Error: 
dsync(lightning.reub.net): I/O has stalled, no activity for 600 seconds 
(last sent=mail, last recv=mail (EOL))
Jun  7 15:11:33 thunderstorm.reub.net dovecot: doveadm: Error: Timeout 
during state=sync_mails (send=mails recv=recv_last_common)


I'm not sure if there is an underlying replication error or if the 
message is just cosmetic, though.


Reuben


On 7/06/2018 4:55 AM, Remko Lodder wrote:

Hi Timo,

Yes this seems to work fine so far. I’ll ask the people to add it to the 
current FreeBSD version..


Cheers
Remko

On 6 Jun 2018, at 19:34, Timo Sirainen > wrote:


Should be fixed by 
https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336

Re: 2.3.1 Replication is throwing scary errors

[Remko Lodder]

Hi Timo,

Yes this seems to work fine so far. I’ll ask the people to add it to the 
current FreeBSD version..

Cheers
Remko

> On 6 Jun 2018, at 19:34, Timo Sirainen  wrote:
> 
> Should be fixed by 
> https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336
>  
> 
> 



signature.asc
Description: Message signed with OpenPGP

Re: Header DATE field not returned when placed after MIME section

[Xavier R. Guérin]

> On Jun 6, 2018, at 1:41 PM, Timo Sirainen  wrote:
> 
> On 2 Jun 2018, at 1.03, Xavier Guerin  wrote:
>> 
>> Hello,
>> 
>> I am using Dovecot 2.2.34p0 on OpenBSD 6.3 stable.
>> 
>> I am writting an application using the vmime (http://www.vmime.org)
>> library (x-ref issue: https://github.com/kisli/vmime/issues/199).
>> 
>> I noticed through testing over my own e-mail server that in some
>> situations the DATE field is not returned upon ENVELOPE or
>> HEADER.FIELDS request.
>> 
>> The e-mails at fault happen to have the DATE field at the very end of
>> the header, after the MIME section.
> 
> Can you easily reproduce this with doveadm? I can't, tested with current 
> master and v2.2.34. :
> 
> % cat << EOF | doveadm save -u tss
> Delivered-To:
> Received:
> Return-Path:
> Delivered-To:
> DKIM-Signature:
> From:
> To:
> Message-ID:
> Subject:
> MIME-Version: 1.0
> Content-Type: text/html;charset=UTF-8
> Content-Transfer-Encoding: 7bit
> Date: Fri, 01 Jun 2018 17:44:44 -0400
> 
> body
> EOF
> 
> % doveadm fetch -u tss hdr.date mailbox inbox '*'
> hdr.date: Fri, 01 Jun 2018 17:44:44 -0400
> 
> % doveadm fetch -u tss imap.envelope mailbox inbox '*'
> imap.envelope: "Fri, 01 Jun 2018 17:44:44 -0400" "" NIL NIL NIL NIL NIL NIL 
> NIL ""
> 
> I tested also with your example 2 mail, but it also worked. Maybe there's 
> something else special in those mails. If you can reproduce it with the above 
> commands, send me the exact mail as attachment that you used?

I reproduced your experiment and it does work. I double checked the raw content 
of the message and it turns out that the Date field has actually been omitted.

I originally used “Evolution” to check the message source and apparently this 
tool shows the Date field even if it is missing.

I believe this situation will have to be handled by the library I am using.

Thanks !

Re: Header DATE field not returned when placed after MIME section

[Timo Sirainen]

On 2 Jun 2018, at 1.03, Xavier Guerin  wrote:
> 
> Hello,
> 
> I am using Dovecot 2.2.34p0 on OpenBSD 6.3 stable.
> 
> I am writting an application using the vmime (http://www.vmime.org)
> library (x-ref issue: https://github.com/kisli/vmime/issues/199).
> 
> I noticed through testing over my own e-mail server that in some
> situations the DATE field is not returned upon ENVELOPE or
> HEADER.FIELDS request.
> 
> The e-mails at fault happen to have the DATE field at the very end of
> the header, after the MIME section.

Can you easily reproduce this with doveadm? I can't, tested with current master 
and v2.2.34. :

% cat << EOF | doveadm save -u tss
Delivered-To:
Received:
Return-Path:
Delivered-To:
DKIM-Signature:
From:
To:
Message-ID:
Subject:
MIME-Version: 1.0
Content-Type: text/html;charset=UTF-8
Content-Transfer-Encoding: 7bit
Date: Fri, 01 Jun 2018 17:44:44 -0400

body
EOF

% doveadm fetch -u tss hdr.date mailbox inbox '*'
hdr.date: Fri, 01 Jun 2018 17:44:44 -0400

% doveadm fetch -u tss imap.envelope mailbox inbox '*'
imap.envelope: "Fri, 01 Jun 2018 17:44:44 -0400" "" NIL NIL NIL NIL NIL NIL NIL 
""

I tested also with your example 2 mail, but it also worked. Maybe there's 
something else special in those mails. If you can reproduce it with the above 
commands, send me the exact mail as attachment that you used?

Re: 2.3.1 Replication is throwing scary errors

[Timo Sirainen]

Should be fixed by 
https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336 

Shared mailboxes, index files and 'per-user-seen' flags

[Thomas Robers]

Hello,

i have a dovecot server version 2.3.1 under CentOS 6.9 and we're
using shared mailboxes with index files shared. With this configuration
I can see a lot of error messages like:

   Jun  6 13:20:31 mail dovecot: Error: imap(us...@tutech.de)<4513>
   : /export/home/imap/us...@tutech.de/shared
   /us...@tutech.de/folder/dovecot.index.pvt view is inconsistent

In 10-mail.conf the location setting is:

   location = maildir:%%h/Maildir:INDEXPVT=%h/shared/%%u

I thought setting the index files to "not shared" might help to
get rid of the errors, so I changed the setting to:

   location = maildir:%%h/Maildir:INDEX=%h/shared/%%u:INDEXPVT=%h
   /shared/%%u

like it's mentioned in the Dovecot wiki. But that doesn't work as
I expected, because the 'per-user-seen' flags do not work correctly
anymore, i think. If UserA, who has UserB as shared mailbox,
changes the seen flags of UserBs INBOX, UserBs seen flags are also
changed. The other way, if UserB changes seen flags in his INBOX
they are not changed in the shared view of UserA. Is this the
supposed way to work  or do i have an error in the configuration?

Any help is appreciated.

Thanks, Thomas.

Here's my currently used configuration:

# 2.3.1 (c5a5c0c82): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.devel (61b47828)
# OS: Linux 2.6.32-696.23.1.el6.x86_64 x86_64 CentOS release 6.9 (Final) 
ext4

# Hostname: mail.tutech.de
auth_master_user_separator = *
auth_mechanisms = plain login
auth_verbose = yes
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
doveadm_port = 12345
imap_max_line_length = 2 M
mail_debug = yes
mail_location = maildir:/export/home/imap/%Lu/Maildir
mail_plugins = acl zlib mail_log notify
mail_prefetch_count = 1
mailbox_idle_check_interval = 10 secs
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext

namespace {
  hidden = no
  ignore_on_failure = no
  inbox = no
  list = children
  location = maildir:%%h/Maildir:INDEXPVT=%h/shared/%%u
  prefix = shared/%%u/
  separator = /
  subscriptions = yes
  type = shared
}
namespace inbox {
  hidden = no
  inbox = yes
  list = yes
  location =
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  type = private
}

passdb {
  args = /etc/dovecot/master-users
  driver = passwd-file
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  acl = vfile:/etc/dovecot/global-acls:cache_secs=300
  acl_shared_dict = file:/export/home/shared-db/shared-mailboxes
  mail_log_events = append delete undelete expunge copy mailbox_delete 
mailbox_rename flag_change

  mail_log_fields = uid box msgid size from flags
  mail_replica = tcp:mail2.tutech.de
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  sieve_global = /var/lib/dovecot/sieve/global/
  sieve_user_log = ~/.dovecot.sieve.log
  zlib_save = gz
  zlib_save_level = 6
}
protocols = imap pop3 lmtp sieve sieve
service aggregator {
  fifo_listener replication-notify-fifo {
mode = 0666
user = vmail
  }
  unix_listener replication-notify {
mode = 0666
user = vmail
  }
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
mode = 0666
  }
  unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
  }
}
service config {
  unix_listener config {
user = vmail
  }
}
service doveadm {
  inet_listener {
port = 12345
  }
  user = vmail
}
service imap-login {
  inet_listener imaps {
port = 993
ssl = yes
  }
  process_limit = 500
  process_min_avail = 20
}
service imap {
  executable = imap
}
service lmtp {
  inet_listener lmtp {
address = 127.0.0.1
port = 24
  }
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  inet_listener sieve_deprecated {
port = 2000
  }
}
service pop3-login {
  inet_listener pop3s {
port = 995
ssl = yes
  }
}
service pop3 {
  executable = pop3
}
service replicator {
  unix_listener replicator-doveadm {
mode = 0666
  }
}
ssl = required
ssl_cert =