Re: doveadm backup from gmail with imapc

[Ben Mulvihill]

Hello again,

I am still stuck I'm afraid.

I now have doveadm backup working perfectly from
a small gmail mailbox (a few hundred messages), but
when I try the same configuration (apart from usernames
and passwords obviously) with a large gmail mailbox
(around 60,000 messages), doveadm connects successfully,
replicates the gmail folder tree, then produces no further
output until about 30 minutes later when the server times
out and doveadm crashes. No actual messages are downloaded. 

Does doveadm backup try by default to download all
messages in a single transaction? And if so, is there
a way to split it up?

Command output and configuration are at the end of this
message. Dovecot version is 2.3.10.

Thanks in advance for any help.

Ben



ben@expectation:~$ doveadm backup -D -R -u ben imapc:
dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
74.125.133.109:993 (local MY_IP_ADDRESS:53244)
dsync(ben): Warning: imapc(imap.gmail.com:993): Server disconnected
with message: System Error - reconnecting (delay 0 ms)
dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
74.125.133.108:993 (local MY_IP_ADDRESS:34076)
dsync(ben): Panic: file imapc-client.c: line 173
(imapc_client_run_pre): assertion failed: (client->ioloop == NULL)
dsync(ben): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f94271610f2] ->
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f942716120e]
-> /usr/lib/dovecot/libdovecot.so.0(+0xe6aee) [0x7f942716baee] ->
/usr/lib/dovecot/libdovecot.so.0(+0xe6b31) [0x7f942716bb31] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f94270c61bb] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imapc_client_run+0x158)
[0x7f94274c27e8] -> /usr/lib/dovecot/libdovecot-
storage.so.0(imapc_simple_run+0x3c) [0x7f94274bf8ac] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x94a23) [0x7f94274b7a23] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_list_get_mailbox_flags+0x68) [0x7f94274b8288] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x98e69) [0x7f94274bbe69] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x98fed) [0x7f94274bbfed] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x9bcc9) [0x7f94274becc9] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xa27cb) [0x7f94274c57cb] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_connection_input_pending+0x227) [0x7f94274c7dc7] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xa4e1a) [0x7f94274c7e1a] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9427184999]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134)
[0x7f9427186274] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c)
[0x7f9427184a9c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f9427184cc8] -> /usr/lib/dovecot/libdovecot-
storage.so.0(imapc_client_run+0xbb) [0x7f94274c274b] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_mailbox_run_nofetch+0x20) [0x7f94274bf7a0] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imapc_mail_fetch+0x199)
[0x7f94274b9419] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9772f)
[0x7f94274ba72f] -> /usr/lib/dovecot/libdovecot-
storage.so.0(mail_get_stream_because+0x64) [0x7f9427461fa4] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(index_mail_update_access_parts_post+0x7a) [0x7f94274e6c2a]
-> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcaf3f) [0x7f94274edf3f]
-> /usr/lib/dovecot/libdovecot-
storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f94274ee1cd]
-> /usr/lib/dovecot/libdovecot-
storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f94274706e2] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d)
[0x7f942747074d] ->
/usr/bin/doveadm(dsync_mailbox_export_next_mail+0x57) [0x562a19bf0597]
Aborted (core dumped)



# 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS
# Hostname: expectation
imapc_features = rfc822.size gmail-migration fetch-headers
imapc_host = imap.gmail.com
imapc_password = # hidden, use -P to show it
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no
imapc_user = MY_GMAIL_ADDRESS
mail_location = maildir:~/Maildir
mail_prefetch_count = 20
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix =
  separator = .
}
passdb {
  driver = pam
}
protocols = " imap"
service stats {
  unix_listener stats-reader {
group = mail
mode = 0660
user = dovecot
  }
  unix_listener stats-writer {
group = mail
mode = 0660
user = dovecot
  }
}
ssl = no
ssl_client_ca_dir = /etc/ssl/certs
userdb {
  driver = passwd
}

Re: Disable Dovecot LDA

[Adam Raszkiewicz]

But then it loops again when get back to the postfix as an incoming message 
(doesn't know that a...@localdomain.com is located on that Dovecot)

 
Adam Raszkiewicz
 
 
p: 845.896.0191 
e: araszkiew...@medallies.com 
w: www.medallies.com  
 
 
 
 
 
 
 
This communication and any files or attachments transmitted with it may contain 
information that is confidential, privileged and exempt from disclosure under 
applicable law. It is intended solely for the use of the individual or the 
entity to which it is addressed. If you are not the intended recipient, you are 
hereby notified that any use, dissemination, or copying of this communication 
is strictly prohibited by federal law. If you have received this communication 
in error, please destroy it and notify the sender.
 

On 3/31/20, 1:45 PM, "dovecot on behalf of Aki Tuomi" 
 wrote:


> On 31/03/2020 19:35 Adam Raszkiewicz  wrote:
> 
> 
>   
> Hi,
>  
>  
> Is there any way to disable Dovecot LDA? I want to always send email via 
postfix and relay server even it will be a local delivery within the Dovecot 
server
>  

Dovecot LDA is fully optional. Just don't use it in postfix configuration

Aki

>  
> Thanks,
>  
> Adam

Re: Disable Dovecot LDA

[Aki Tuomi]

> On 31/03/2020 19:35 Adam Raszkiewicz  wrote:
> 
> 
>   
> Hi,
>  
>  
> Is there any way to disable Dovecot LDA? I want to always send email via 
> postfix and relay server even it will be a local delivery within the Dovecot 
> server
>  

Dovecot LDA is fully optional. Just don't use it in postfix configuration

Aki

>  
> Thanks,
>  
> Adam

Disable Dovecot LDA

[Adam Raszkiewicz]

Hi,

Is there any way to disable Dovecot LDA? I want to always send email via 
postfix and relay server even it will be a local delivery within the Dovecot 
server

Thanks,
Adam

net_connect_unix(imap) failed

[Philipp Ewald]

Hello everyone,

we have a huge problem with dovecot and IMAP connections.

we got the following errors: 39665 today

dovecot: imap-login: Error: master(imap): net_connect_unix(imap) failed: 
Resource temporarily unavailable - http://wiki2.dovecot.org/SocketUnavailable 
(client-pid=29066, client-id=1755, rip=IP, created 562 msecs ago, received 0/4 
bytes)

we thinks this may be a problem with authorization take too long? Authorization 
is not local and with SQL.

i found the following in source code of Dovecot:
#define SOCKET_CONNECT_RETRY_MSECS 500
#define SOCKET_CONNECT_RETRY_WARNING_INTERVAL_SECS 2
[...]
i_error("master(%s): %s (client-pid=%u, client-id=%u, rip=%s, created %u msecs ago, 
received %u/%zu bytes)",

This is no process limit problem:

ps auxf | grep -c "[d]ovecot/imap$"
688

ps auxf | grep -c "[d]ovecot/imap-login$"
100


cat /proc/`pidof dovecot`/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size8388608  unlimitedbytes
Max core file size0unlimitedbytes
Max resident set  unlimitedunlimitedbytes
Max processes 6405364053processes
Max open files6553565535files
Max locked memory 6553665536bytes
Max address space unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max pending signals   6405364053signals
Max msgqueue size 819200   819200   bytes
Max nice priority 00
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus

ulimit -n
1024


dovecot --version
2.3.4.1 (f79e8e7e4)


protocols = imap pop3
service imap-login {
  process_min_avail = 4
  service_count = 0
}
service imap {
  process_limit = 4096
}
service pop3-login {
  process_min_avail = 4
  service_count = 0
}
service pop3 {
  process_limit = 4096
}



Can someone explain why we got this error and how to fix? If you need another 
information please tell me.


--
Philipp Ewald
Administrator

DigiOnline GmbH, Probsteigasse 15 - 19, 50670 Köln
E-Mail: philipp.ew...@digionline.de

Web: www.digionline.de, www.webweaver.de

AG Köln HRB 27711, St.-Nr. 5215 5811 0640
Geschäftsführer: Werner Grafenhain

Informationen zum Datenschutz: www.digionline.de/ds

Re: limit for user exceeded

[Maciej Milaszewski]

Hi
I Dont understand or  I im thinking wrong:

process_limit = 25000


Older:

#fs.inotify.max_user_watches= 8192
#fs.inotify.max_user_instances = 16384

New:
fs.inotify.max_user_instances = 8192
 
fs.inotify.max_user_watches= process_limit x 2 + fs.inotify.max_user_instances
fs.inotify.max_user_watches= 58192


On 31.03.2020 13:44, Aki Tuomi wrote:
> I would prefer replies on the list... =)
>
> no. the idea is to *increase* the *current* value of 
> fs.inotify.max_user_watches and fs.inotify.max_user_instances with 5
>
> fs.inotify.max_user_watches = 8192 + 5 = 58192
>
> Aki
>
>> On 31/03/2020 14:21 Maciej Milaszewski  wrote:
>>
>>  
>> Hi
>> How I understood it correctly
>>
>> service imap {
>>   process_limit = 25000
>> }
>>
>> fs.inotify.max_user_watches= 5
>> fs.inotify.max_user_instances = 5
>>
>> ?
>>
>>
>> On 31.03.2020 12:14, Aki Tuomi wrote:
>>> Sorry, ment that we *increase* the current value with twice the process 
>>> limit for IMAP.
>>>
>>> Aki
>>>
 On 31/03/2020 13:12 Aki Tuomi  wrote:

  
 We usually set them to twice the number of process_limit for imap.

 Aki

> On 31/03/2020 12:29 Maciej Milaszewski  wrote:
>
>  
> Hi
> System debian 8.11 dovecot-2.2.36.4 and I have some warnings in log likes:
>
> Warning: Inotify watch limit for user exceeded, disabling. Increase
> /proc/sys/fs/inotify/max_user_watches
>
>
> cat /proc/sys/fs/inotify/max_user_watches
> 8192
>
> in sysctl i change
>
> #fs.inotify.max_user_watches= 8192
> #fs.inotify.max_user_instances = 16384
>
> fs.inotify.max_user_watches= 16384
> fs.inotify.max_user_instances = 24576
>
> One questions - should these values be equal?

Re: doveadm backup from gmail with imapc

[Ben Mulvihill]

On Tue, 2020-03-31 at 09:06 +0300, Aki Tuomi wrote:
> > 
> > On 30/03/2020 22:11 Ben Mulvihill  wrote:
> > 
> >  
> > I am trying to backup a gmail account (not the one I am writing
> > from)
> > to dovecot, using doveadm-backup and imapc, but am having ssl
> > connection problems.
> > 
> > ted@expectation:~# doveadm backup -D -R -u ted imapc:
> > dsync(ted): Info: imapc(imap.gmail.com:993): Connected to
> > 74.125.71.108:993 (local 10.7.1.179:53852)
> > dsync(ted): Warning: imapc(imap.gmail.com:993): Server disconnected
> > unexpectedly: SSL_connect() failed: error:14094410:SSL
> > routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert
> > number 40 - reconnecting (delay 0 ms)
> > dsync(ted): Info: imapc(imap.gmail.com:993): Connected to
> > 74.125.71.109:993 (local 10.7.1.179:59052)
> > dsync(ted): Error: imapc(imap.gmail.com:993): Server disconnected
> > unexpectedly: SSL_connect() failed: error:14094410:SSL
> > routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert
> > number 40 - disconnecting
> > dsync(ted): Error: User initialization failed: imapc: Login to
> > imap.gmail.com failed: Disconnected from serv
> > 
> > 
> > I am using dovecot version 2.2.33.2 on ubuntu, with the
> > configuration below.
> > I have also enabled "allow access from unsecure apps" in my
> > gmail settings.
> > 
> > My first thought looking at the error messages was has that
> > perhaps doveadm-backup was trying to connect with ssl3, which
> > is no longer supported by gmail or anyone else nowadays.
> > But apparently the ssl3_read_bytes function in openssl also
> > handles tls, so the reference to ssl3 in the message is
> > misleading, and the real problem must be elsewhere.
> > 
> > If anyone can help me debug this I'd be grateful.
> > 
> > Many thanks, 
> > Ben
> > 
> > root@expectation:/etc/dovecot# doveconf -N
> > # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
> Hi!
> 
> This is very old version of dovecot so this could be a bug that has
> been fixed in more recent version.
> 
> Can you verify that you have the required CA certs with
> 
> openssl s_client -connect imap.gmail.com:993 -servername
> imap.gmail.com -CApath /etc/ssl/certs
> 
> and make sure the cert gets validated by openssl.
> 
> If it does, then you should probably consider upgrading to some more
> recent version. We provide packages at https://repo.dovecot.org if
> you are able to upgrade.
> 
> Aki

Hi Aki!

I checked the CA certificate and it was validated OK, so I
followed your suggestion and installed dovecot version 2.3.10
from the dovecot repository (the version I was using previously
was the latest available in my distribution).

The ssl handshake now completes successfully. I have hit 
further problems later in the process, but I'll investigate
myself first and then maybe post again with the details if
I am still stuck.


Thank you for your help!

Ben

Re: mail_crypt crashing randomly during LMTP

[Timo Sirainen]

On 31. Mar 2020, at 13.47, Timo Sirainen  wrote:
> 
> On 29. Mar 2020, at 17.07, Timo Sirainen  wrote:
>> 
>>> This is what the log says in the moment of crashing:
>>> dovecot: lmtp(53852): Panic: Module context mail_crypt_user_module missing
>>> *** backtrace *** (see: https://pastebin.com/YCiFtxmy)
>> 
>> It shows mailbox_free() in the backtrace, so it's probably crashing in 
>> mail_crypt_mailbox_close()'s MAIL_CRYPT_CONTEXT(box). But looking at the 
>> code, I can't see how that could be possible. Maybe there is some kind of 
>> memory corruption or something.
> 
> 
> Looks like there's a bug when some users have mail_crypt plugin loaded and 
> other users don't. If LMTP delivers a mail to both users within the same 
> session, it crashes at deinit. A workaround could be to tell MTA to deliver 
> to max recipient at a time. We'll get this fixed also to some future release. 
> You could also apply a patch:

Looks like there is also a better way to do this. Enable mail_crypt plugin 
always in dovecot.conf, but return mail_crypt_save_version differently for 
users. So mail_crypt_save_version=0 for users who don't want encryption, 
mail_crypt_save_version=2 for users who do want encryption.

Re: mail_crypt crashing randomly during LMTP

[Timo Sirainen]

On 29. Mar 2020, at 17.07, Timo Sirainen  wrote:
> 
>> This is what the log says in the moment of crashing:
>> dovecot: lmtp(53852): Panic: Module context mail_crypt_user_module missing
>> *** backtrace *** (see: https://pastebin.com/YCiFtxmy)
> 
> It shows mailbox_free() in the backtrace, so it's probably crashing in 
> mail_crypt_mailbox_close()'s MAIL_CRYPT_CONTEXT(box). But looking at the 
> code, I can't see how that could be possible. Maybe there is some kind of 
> memory corruption or something.


Looks like there's a bug when some users have mail_crypt plugin loaded and 
other users don't. If LMTP delivers a mail to both users within the same 
session, it crashes at deinit. A workaround could be to tell MTA to deliver to 
max recipient at a time. We'll get this fixed also to some future release. You 
could also apply a patch:

diff --git a/src/plugins/mail-crypt/mail-crypt-plugin.c 
b/src/plugins/mail-crypt/mail-crypt-plugin.c
index 43ece3d3b..a37fb4a35 100644
--- a/src/plugins/mail-crypt/mail-crypt-plugin.c
+++ b/src/plugins/mail-crypt/mail-crypt-plugin.c
@@ -345,7 +345,7 @@ static void mail_crypt_mailbox_close(struct mailbox *box)
 {
struct mail_crypt_mailbox *mbox = MAIL_CRYPT_CONTEXT(box);
struct mail_crypt_user *muser =
-   MAIL_CRYPT_USER_CONTEXT_REQUIRE(box->storage->user);
+   MAIL_CRYPT_USER_CONTEXT(box->storage->user);

if (mbox->pub_key != NULL)
dcrypt_key_unref_public(>pub_key);

Re: limit for user exceeded

[Aki Tuomi]

Sorry, ment that we *increase* the current value with twice the process limit 
for IMAP.

Aki

> On 31/03/2020 13:12 Aki Tuomi  wrote:
> 
>  
> We usually set them to twice the number of process_limit for imap.
> 
> Aki
> 
> > On 31/03/2020 12:29 Maciej Milaszewski  wrote:
> > 
> >  
> > Hi
> > System debian 8.11 dovecot-2.2.36.4 and I have some warnings in log likes:
> > 
> > Warning: Inotify watch limit for user exceeded, disabling. Increase
> > /proc/sys/fs/inotify/max_user_watches
> > 
> > 
> > cat /proc/sys/fs/inotify/max_user_watches
> > 8192
> > 
> > in sysctl i change
> > 
> > #fs.inotify.max_user_watches= 8192
> > #fs.inotify.max_user_instances = 16384
> > 
> > fs.inotify.max_user_watches= 16384
> > fs.inotify.max_user_instances = 24576
> > 
> > One questions - should these values be equal?

Re: limit for user exceeded

[Aki Tuomi]

We usually set them to twice the number of process_limit for imap.

Aki

> On 31/03/2020 12:29 Maciej Milaszewski  wrote:
> 
>  
> Hi
> System debian 8.11 dovecot-2.2.36.4 and I have some warnings in log likes:
> 
> Warning: Inotify watch limit for user exceeded, disabling. Increase
> /proc/sys/fs/inotify/max_user_watches
> 
> 
> cat /proc/sys/fs/inotify/max_user_watches
> 8192
> 
> in sysctl i change
> 
> #fs.inotify.max_user_watches= 8192
> #fs.inotify.max_user_instances = 16384
> 
> fs.inotify.max_user_watches= 16384
> fs.inotify.max_user_instances = 24576
> 
> One questions - should these values be equal?

limit for user exceeded

[Maciej Milaszewski]

Hi
System debian 8.11 dovecot-2.2.36.4 and I have some warnings in log likes:

Warning: Inotify watch limit for user exceeded, disabling. Increase
/proc/sys/fs/inotify/max_user_watches


cat /proc/sys/fs/inotify/max_user_watches
8192

in sysctl i change

#fs.inotify.max_user_watches= 8192
#fs.inotify.max_user_instances = 16384

fs.inotify.max_user_watches= 16384
fs.inotify.max_user_instances = 24576

One questions - should these values be equal?

Re: doveadm backup from gmail with imapc

[Aki Tuomi]

> On 30/03/2020 22:11 Ben Mulvihill  wrote:
> 
>  
> I am trying to backup a gmail account (not the one I am writing from)
> to dovecot, using doveadm-backup and imapc, but am having ssl
> connection problems.
> 
> ted@expectation:~# doveadm backup -D -R -u ted imapc:
> dsync(ted): Info: imapc(imap.gmail.com:993): Connected to
> 74.125.71.108:993 (local 10.7.1.179:53852)
> dsync(ted): Warning: imapc(imap.gmail.com:993): Server disconnected
> unexpectedly: SSL_connect() failed: error:14094410:SSL
> routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert
> number 40 - reconnecting (delay 0 ms)
> dsync(ted): Info: imapc(imap.gmail.com:993): Connected to
> 74.125.71.109:993 (local 10.7.1.179:59052)
> dsync(ted): Error: imapc(imap.gmail.com:993): Server disconnected
> unexpectedly: SSL_connect() failed: error:14094410:SSL
> routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert
> number 40 - disconnecting
> dsync(ted): Error: User initialization failed: imapc: Login to
> imap.gmail.com failed: Disconnected from serv
> 
> 
> I am using dovecot version 2.2.33.2 on ubuntu, with the
> configuration below.
> I have also enabled "allow access from unsecure apps" in my
> gmail settings.
> 
> My first thought looking at the error messages was has that
> perhaps doveadm-backup was trying to connect with ssl3, which
> is no longer supported by gmail or anyone else nowadays.
> But apparently the ssl3_read_bytes function in openssl also
> handles tls, so the reference to ssl3 in the message is
> misleading, and the real problem must be elsewhere.
> 
> If anyone can help me debug this I'd be grateful.
> 
> Many thanks, 
> Ben
> 
> root@expectation:/etc/dovecot# doveconf -N
> # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf

Hi!

This is very old version of dovecot so this could be a bug that has been fixed 
in more recent version.

Can you verify that you have the required CA certs with

openssl s_client -connect imap.gmail.com:993 -servername imap.gmail.com -CApath 
/etc/ssl/certs

and make sure the cert gets validated by openssl.

If it does, then you should probably consider upgrading to some more recent 
version. We provide packages at https://repo.dovecot.org if you are able to 
upgrade.

Aki