Re: Dbox lost messages

[Fineware PM]

Hello, 

I try what you write, nothing change, 

I try to delete the index file under user/idx/mailboxes/INBOX/ [1] 

and the re-run doveadm force-resync -u user "*" : nothing change. 

Is a mistery, I can send the m. file to someone to understand what is
the problem ?

Il 20/07/2018 14:57 Sami Ketola ha scritto:

>> On 20 Jul 2018, at 15.20, Fineware PM  wrote: 
>> 
>> if I dump the index of inbox : 
>> "doveadm -v dump ATTENZIONE, LIBRA ESVA HA RILEVATO UN SITO CLASSIFICATO 
>> COME PHISHING: "EXAMPLE.COM". NON FIDARSI DI QUESTO SITO: 
>> example.com/user/idx/mailboxes/INBOX/dovecot.index [1]"  
>> 
>> I don't find the message. 
>> 
>> I think is corrupted the index
> 
> try then running: doveadm force-resync -u user "*" 
> Sami 
> 
> -- 
> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non 
> infetto. :)
> Clicca qui per segnalarlo come spam. [2] 
> Clicca qui per metterlo in blacklist [3]
 

Links:
--
[1] http://example.com/user/idx/mailboxes/INBOX/dovecot.index
[2] https://esva.fineware.it/cgi-bin/learn-msg.cgi?id=B96674077E.A2772
[3]
https://esva.fineware.it/cgi-bin/learn-msg.cgi?blacklist=1&id=B96674077E.A2772

Re: Dbox lost messages

[Sami Ketola]

> On 20 Jul 2018, at 15.20, Fineware PM  wrote:
> 
> if I dump the index of inbox :
> "doveadm -v dump example.com/user/idx/mailboxes/INBOX/dovecot.index" 
>  
> I don't find the message.
> I think is corrupted the index
> 
> 


try then running: doveadm force-resync -u user "*"

Sami

Re: Dbox lost messages

[Fineware PM]

I make other test: 

If I dump the message file m.xxx with doveadm dump I found 

Detected file type: dbox
file.msg_header_size = 30
file.create_stamp = 1532017559 (2018-07-19 18:25:59)

offset 16:
msg.size = 2252089
msg.received = 1532017559 (2018-07-19 18:25:59)
msg.virtual-size = 2281418
msg.guid = 06b7dd2c97bb505b5434193e0904
msg.orig-mailbox = INBOX

if I dump the index of inbox : 
"doveadm -v dump example.com/user/idx/mailboxes/INBOX/dovecot.index"  

I don't find the message. 

I think is corrupted the index 

Il 20/07/2018 14:04 Fineware PM ha scritto:

> Hello, 
> 
> no sieve SCRIPT. 
> 
> in the maillog say "Stored to INBOX". 
> 
> I don't have the maillog plugin, I install it now to track future problems. 
> 
> But If I backup the inbox all email (also that I cannot see) is in the mbox 
> file.
> 
> Il 20/07/2018 13:56 Sami Ketola ha scritto: 
> 
> On 20 Jul 2018, at 14.48, Fineware PM  wrote: 
> 
> Hello, 
> 
> I use dovecot with dbox storage system. 
> 
> Today I receive 2 mail (I see that in the maillog ) but I don't find in INBOX 
> using my IMAP client. 
> 
> I look in the files m.* and I find the message that I receive. But I cannot 
> find it in any IMAP folder. 
> 
> I try to run "doveadm force-resync -u user inbox" but nothing change. 
> 
> I export all email using the command "doveadm backup -u user 
> mbox:/tmp/mail:INDEX:/tmp/imap:LAYOUT:maildir++ 
> 
> and in the mbox file for inbox I found the 2 mails arrived. 
> 
> How I can understand why I cannot see the messages ? or how I can recover it 
> ? 
> 
> Bet regards.
> 
> Can you see anything in the logs related? Do you use sieve scripts? Does the 
> user have sieve script that might have moved/deleted the message? 
> Do you have maillog plugin enabled to log what the users do? Can you see the 
> user deleting the mails? https://wiki.dovecot.org/Plugins/MailLog 
> 
> Sami 
> 
> -- 
> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non 
> infetto. :)
> Clicca qui per segnalarlo come spam. [1] 
> Clicca qui per metterlo in blacklist [2]
 

Links:
--
[1] https://esva.fineware.it/cgi-bin/learn-msg.cgi?id=383CA40774.A35FA
[2]
https://esva.fineware.it/cgi-bin/learn-msg.cgi?blacklist=1&id=383CA40774.A35FA

Re: Dbox lost messages

[Fineware PM]

Hello, 

no sieve SCRIPT. 

in the maillog say "Stored to INBOX". 

I don't have the maillog plugin, I install it now to track future
problems. 

But If I backup the inbox all email (also that I cannot see) is in the
mbox file.

Il 20/07/2018 13:56 Sami Ketola ha scritto:

>> On 20 Jul 2018, at 14.48, Fineware PM  wrote: 
>> 
>> Hello, 
>> 
>> I use dovecot with dbox storage system. 
>> 
>> Today I receive 2 mail (I see that in the maillog ) but I don't find in 
>> INBOX using my IMAP client. 
>> 
>> I look in the files m.* and I find the message that I receive. But I cannot 
>> find it in any IMAP folder. 
>> 
>> I try to run "doveadm force-resync -u user inbox" but nothing change. 
>> 
>> I export all email using the command "doveadm backup -u user 
>> mbox:/tmp/mail:INDEX:/tmp/imap:LAYOUT:maildir++ 
>> 
>> and in the mbox file for inbox I found the 2 mails arrived. 
>> 
>> How I can understand why I cannot see the messages ? or how I can recover it 
>> ? 
>> 
>> Bet regards.
> 
> Can you see anything in the logs related? Do you use sieve scripts? Does the 
> user have sieve script that might have moved/deleted the message? 
> Do you have maillog plugin enabled to log what the users do? Can you see the 
> user deleting the mails? https://wiki.dovecot.org/Plugins/MailLog 
> 
> Sami 
> 
> -- 
> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non 
> infetto. :)
> Clicca qui per segnalarlo come spam. [1] 
> Clicca qui per metterlo in blacklist [2]
 

Links:
--
[1] https://esva.fineware.it/cgi-bin/learn-msg.cgi?id=383CA40774.A35FA
[2]
https://esva.fineware.it/cgi-bin/learn-msg.cgi?blacklist=1&id=383CA40774.A35FA

Re: Dbox lost messages

[Sami Ketola]

> On 20 Jul 2018, at 14.48, Fineware PM  wrote:
> 
> Hello,
> 
> I use dovecot with dbox storage system.
> 
> Today I receive 2 mail (I see that in the maillog ) but I don't find in INBOX 
> using my IMAP client.
> 
> I look in the files m.* and I find the message that I receive. But I cannot 
> find it in any IMAP folder.
> 
> I try to run "doveadm force-resync -u user inbox" but nothing change.
> 
> I export all email using the command "doveadm backup -u user 
> mbox:/tmp/mail:INDEX:/tmp/imap:LAYOUT:maildir++
> 
> and in the mbox file for inbox I found the 2 mails arrived.
> 
> How I can understand why I cannot see the messages ? or how I can recover it ?
> 
> Bet regards.
> 
> 

Can you see anything in the logs related? Do you use sieve scripts? Does the 
user have sieve script that might have moved/deleted the message?
Do you have maillog plugin enabled to log what the users do? Can you see the 
user deleting the mails? https://wiki.dovecot.org/Plugins/MailLog 


Sami

Re: OT: 'lost' emails

[Sami Ketola]

> On 20 Jul 2018, at 13.31, Voytek  wrote:
> 
> I suspect this is mail client issue, but looking for any suggestions:
> 
> user with Thunderbird says " I'm losing emails, mail is not in inbox, but, 
> when I search, that email shows in search result, but, I can't open the email 
> body" i don't fully understand what he was trying to say, and, won't be able 
> to look at his PC till Monday.
> 
> Looking at inbox's 'cur' directory, I can see the'missing' email amongst the 
> other 2000 emails in 'cur'
> 
> Is there any indexing I should do on the user's email folder, how?
> 
> or, is this purely Thunderbird issue, and, I need to re index Thunderbird?

https://support.mozilla.org/it/questions/1050806

seems so.

Sami

Dbox lost messages

[Fineware PM]

Hello, 

I use dovecot with dbox storage system. 

Today I receive 2 mail (I see that in the maillog ) but I don't find in
INBOX using my IMAP client. 

I look in the files m.* and I find the message that I receive. But I
cannot find it in any IMAP folder. 

I try to run "doveadm force-resync -u user inbox" but nothing change. 

I export all email using the command "doveadm backup -u user
mbox:/tmp/mail:INDEX:/tmp/imap:LAYOUT:maildir++ 

and in the mbox file for inbox I found the 2 mails arrived. 

How I can understand why I cannot see the messages ? or how I can
recover it ? 

Bet regards.

Re: OT: 'lost' emails

[Michael A. Peters]

On 07/20/2018 03:31 AM, Voytek wrote:

I suspect this is mail client issue, but looking for any suggestions:

user with Thunderbird says " I'm losing emails, mail is not in inbox,
but, when I search, that email shows in search result, but, I can't open
the email body" i don't fully understand what he was trying to say, and,
won't be able to look at his PC till Monday.

Looking at inbox's 'cur' directory, I can see the'missing' email amongst
the other 2000 emails in 'cur'

Is there any indexing I should do on the user's email folder, how?

or, is this purely Thunderbird issue, and, I need to re index Thunderbird?

V
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.


I'm using 45.8.0 because I experienced weird things with newer (CentOS) 
such as inability to use self-signed certs properly, but I didn't 
experience that issue.


Not sure if that was CentOS 7 issue or Thunderbird, but the newer 
versions did give me trouble.


That issue though sounds like corrupted client database, not sure if 
there is a way to rebuild it but I suspect there is.


I experienced similar issues frequently in evolution, hence why I 
switched to thunderbird.

OT: 'lost' emails

[Voytek]

I suspect this is mail client issue, but looking for any suggestions:

user with Thunderbird says " I'm losing emails, mail is not in inbox, but, when 
I search, that email shows in search result, but, I can't open the email body" 
i don't fully understand what he was trying to say, and, won't be able to look 
at his PC till Monday.

Looking at inbox's 'cur' directory, I can see the'missing' email amongst the 
other 2000 emails in 'cur'

Is there any indexing I should do on the user's email folder, how?

or, is this purely Thunderbird issue, and, I need to re index Thunderbird?

V
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

dovecot sometimes sends non-default SSL cert if IMAP client won't send SNI

[Martin Johannes Dauser]

Hi,

I recognised some funny behaviour on my server. IMAP clients which
won't send an Server Name Indication (SNI) sometimes get the wrong
certificate. I would expect that those clients always get the default
certificate (of my new domain), instead in about 20 to 50% of
connections the certificate of my old domain will be presented.
(sample rate was 3 times 30 connections)

Clients sending SNI always get the right certificate.

A user informed me that offlineIMAP complains 
'CA Cert verifying failed:
   no matching domain name found in certificate'
So at least offlineIMAP 7.0.12 from Debain stretch won't send SNI,
there is a newer version upstream though.


I myself checked the server's behaviour with openssl:

$ openssl s_client -showcerts -connect IP-address:993

and

$ openssl s_client -showcerts -connect IP-address:993 -servername
imap.domain


I'm totally clueless about how come.

Best regards
Martin Johannes Dauser




# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-862.el7.x86_64 x86_64 Red Hat Enterprise Linux
Server release 7.5 (Maipo) 

...

service imap-login {
  inet_listener imap {
address = 127.0.0.1
port = 143
  }
  inet_listener imaps {
port = 993
ssl = yes
  }
  process_min_avail = 8
  service_count = 0
}

...

ssl = required
# set default cert
ssl_cert = 

Re: SIS: dovecot creates duplicated attachments (sometimes)

[Alexander Moisseev]

It's been 2 years since my previous post, but the issue is still present 
(Dovecot 2.3.2).

Long story short, SIS attachment deduplication was broken in Dovecot v2.2.16. 
When it was fixed in v2.2.22 I wrote this script to deduplicate attachments on 
my servers back: 
https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck
Sometimes I run the script and it finds new duplicated attachments (no too 
many). If you check these attachments, you find out that the attachments has 
the same hash, the same size, match if compare them byte by byte, were created 
at the same time, but one of them has a different inode number.

For instance, 3 days ago it was:
  Attachments processed: 31535
  Duplicated attachments: 370
  Missed hash files: 1313

And today:
  Attachments processed: 32368
  Duplicated attachments: 381
  Missed hash files: 1317

So, 11 new duplicates and 4 new missed hashes.

Can someone suggest a way to debug this problem somehow?

As a lot of time has passed since my previous post, I repost it below.

On 5/16/2016 9:27 AM, Alexander Moisseev wrote:

SIS deduplication was broken in 2.2.16 and has been fixed with 
https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1
but still sometimes (just several times in a month or so) dovecot creates 
duplicated attachments.

As you can see in directory listings below all attachments was created at 
(about) the same time, but one of them has different inode number.


It easy to discover using 
https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck

# dsisck -n

mail_uid=vmail
mail_attachment_dir=/vmail/attachments

==> Checking SIS...

# ln -f 
./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd
 
./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd
# mkdir -p -m 700 ./64/8f/hashes && ln 
./64/8f/648f5cfa27af6d20c8570fdcaeab997663e15105-55430d2cd4432c571cb63d96bafd 
./64/8f/hashes/648f5cfa27af6d20c8570fdcaeab997663e15105
# ln -f 
./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd
 
./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd
# ln -f 
./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd
 
./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd

--
  Unexpected objects found: 0

  Attachments processed: 43500
  Different attachments with similar hashes skipped: 0

  Attachments deduplicated: 3
  File system blocks freed up: 1800
  Attachment deduplication attempts failed: 0

  Hash files created: 1
  Hash files creation attempts failed: 0

  Hash files re-linked: 0
  Hash files re-link attempts failed: 0

  Orphaned hash files: 0
  Invalid hash files (not a regular file): 0
--

# cd /vmail/attachments


# ls -li ./30/b3/ ./30/b3/hashes/
./30/b3/:
total 1996
7464083 -rw---  1 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bb6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bd6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bf6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-c36eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd
7464084 drwx--  2 vmail  vmail 512 May 10 09:16 hashes

./30/b3/hashes/:
total 332
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545

# ls -li ./6f/3f/ ./6f/3f/hashes/
./6f/3f/:
total 644
15088414 -rw---  1 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd
15088422 -rw---  2 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd
15088448 drwx--  2 vmail  vmail 512 May 13 11:23 hashes

./6f/3f/hashes/:
total 320
15088422 -rw---  2 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe

# ls -li ./dd/d9/ ./dd/d9/hashes/
./dd/d9/:
total 748
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-68f7c122fefb3157ad353d96bafd
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd
80547 -rw---  1 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd
80549 drwx--  2 vmail  vmail 512 May 10 18:19 hashes

./dd/d9/hash