index problems after update

[Ajay Guleria]

I Upgraded a server from Ubuntu 16.04 to Ubuntu 18.04. This includes an upgrade 
from dovecot 2.2.22 to 2.2.33.2. 
Sometimes it happens, that dovecot stops showing new mail. There is no error in 
the log, dovecot just seems to do his normal operations but is not delivering 
new emails from /var/mail/user (for some users only) emails are delivered again 
if I delete index-files from .imap folder.
Ajay Guleria

Re: index problems after update

[Hajo Locke via dovecot]

Hello,


Am 21.02.2019 um 23:06 schrieb Adi Pircalabu via dovecot:

On 2019-02-21 22:18, Sami Ketola via dovecot wrote:

On 21 Feb 2019, at 12.23, Hajo Locke via dovecot
 wrote:
I think mbox+procmail is a classic setup and wide used and good
solution for many usecases. Same setup we use many years.
We run ~2 mio mailboxes. our automated systems depends on this
setup. creating mailboxes, managing mailboxes, creating automated
filterrules, backupsystem to tell something of them. we can not
switch our whole mailsetup to work around this bug.
How to get a dump if dovecot not crashing but has wrong behaviour? I
would like to help and provide useful info, but it depends on kind
of problem.
I think if a classic setup is not working in dovecot any more, this
is a serious problem.


In you first email to this thread it says:


Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal:
master: service(imap): child 14135 killed with signal 6 (core dumped)


So imap is crashing and even dumping a core.

Also I must disagree with your mbox+procmail statement. mbox has
always been very unoptimised mailbox format and everyone should be
emphasised not to use it.
Also that combination has always had problems with indexing and file
locking. I would not use it on high volume mailservers. Or even medium
volume mailservers.


Not directly affected by this issue since I'm not using mbox for any
production system nor have I for many years. And it'd take a lot of
effort to convince me to use mbox for anything someone would even dare
to classify, even remotely, as "production". But if I understand OP's
point of view correctly, he's not arguing necessarily for or against a
specific mailbox format. Instead, he's flagging a regression and
people will be very reluctant to upgrade or even adopt a certain
feature in a new release of a product if regressions are seen as
acceptable. Something that previously worked in an otherwise unchanged
environment stopped working after an upgrade and this is a regression.
Trying to convince people to move away from mbox is a very sensible
approach, I'm all for it, but in cases like this not practical.


yes, thanks for your support. I think this is the case here.

As conclusion:
We believe mdbox is better storage format, but we cant switch 2mio
mboxes + peripherie adhoc.
We did a downgrade to 2.2.26.0 a few weeks ago. 2.2.26.0 is first
version usable with openssl 1.1.0 and solves this issue. so this dovecot
problem must introduced after 2.2.26.0 and leads to a problem where
dovecot is not able to fulfill its basic purpose.
until this downgrade we had a lot of problems and complaints and felt
forsaken.
Too bad that there is no really interest to fix mbox related problems.

Thanks,
Hajo

Re: index problems after update

[Adi Pircalabu via dovecot]

On 2019-02-21 22:18, Sami Ketola via dovecot wrote:
On 21 Feb 2019, at 12.23, Hajo Locke via dovecot  
wrote:
I think mbox+procmail is a classic setup and wide used and good 
solution for many usecases. Same setup we use many years.
We run ~2 mio mailboxes. our automated systems depends on this setup. 
creating mailboxes, managing mailboxes, creating automated 
filterrules, backupsystem to tell something of them. we can not switch 
our whole mailsetup to work around this bug.
How to get a dump if dovecot not crashing but has wrong behaviour? I 
would like to help and provide useful info, but it depends on kind of 
problem.
I think if a classic setup is not working in dovecot any more, this is 
a serious problem.


In you first email to this thread it says:

Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
service(imap): child 14135 killed with signal 6 (core dumped)


So imap is crashing and even dumping a core.

Also I must disagree with your mbox+procmail statement. mbox has
always been very unoptimised mailbox format and everyone should be
emphasised not to use it.
Also that combination has always had problems with indexing and file
locking. I would not use it on high volume mailservers. Or even medium
volume mailservers.


Not directly affected by this issue since I'm not using mbox for any 
production system nor have I for many years. And it'd take a lot of 
effort to convince me to use mbox for anything someone would even dare 
to classify, even remotely, as "production". But if I understand OP's 
point of view correctly, he's not arguing necessarily for or against a 
specific mailbox format. Instead, he's flagging a regression and people 
will be very reluctant to upgrade or even adopt a certain feature in a 
new release of a product if regressions are seen as acceptable. 
Something that previously worked in an otherwise unchanged environment 
stopped working after an upgrade and this is a regression. Trying to 
convince people to move away from mbox is a very sensible approach, I'm 
all for it, but in cases like this not practical.


--
Adi Pircalabu

Re: index problems after update

[Gerald Galster via dovecot]

For replicated servers I'm stuck with 2.2.33.2 because of pop3/dsync problems,
but on single servers I have no index problems after upgrading to 
dovecot-2.2.35-1.el7.centos.0.x86_64
or dovecot-2.2.36-3.el7.x86_64.

All servers run CentOS 7 (RHEL 7) but use lmtp delivery with mdbox and sieve.
Maybe something in dovecot-lda has changed?

Best regards
Gerald

> Am 21.02.2019 um 14:12 schrieb Gonzalo Palacios Goicolea via dovecot 
> :
> 
> El 21/02/2019 a las 10:51, Aki Tuomi via dovecot escribió:
>> On 21.2.2019 10.53, Hajo Locke via dovecot wrote:
>>> Hello,
>>> 
>>> Am 20.02.2019 um 10:39 schrieb Aki Tuomi via dovecot:
> On 18 February 2019 09:28 Hajo Locke via dovecot
>   wrote:
> 
> 
> Hello,
> it seems we need a dovecot developers opinion. May be we hit a
> bug or cant help ourselves.
>
>>> Thanks for your answer.
 Core dump with backtrace would help, if possible to acquire. Please
 refer to https://dovecot.org/bugreport.html 
  for information how to
 get a core dump.
 
 Aki
 
>>> Unfortunately its hard to get a backtrace because dovecot is not
>>> crashing. so it seems to be more a kind of logic problem in code and
>>> no unexpected situation.
>>> yesterday evening i had next incident. I upgraded from 2.2.33.2 to
>>> 2.2.36.1, but same behaviour. Also 2.2.36.1 is tricked by the broken
>>> index and delivers no new mails. it starts delivering if i delete
>>> index files. At this point i cant tell if 2.2.36.1 also has same bug
>>> and writes a damaged index, but very likely.  We dont know this
>>> problems with 2.2.22, between 2.2.22 and 2.2.33.2 a change on
>>> mbox-index code must happend which leads to this big problem. So imapd
>>> cant do what he was created for.
>>> 
>>> For next incident i prepared a 2.3.2.1 on base of Ubuntu 18.10 and
>>> will try this. In my opinion this is a major problem and i expect a
>>> lot of affected people with version > 2.2.22 and classic mbox-storage.
>>> 
>>> Thanks,
>>> Hajo
>> We consider mbox + procmail setup somewhat edge case, and if the core
>> dump does not point into something more generic, it will probably not
>> get fixed. It is more likely to have this working if you use
>> dovecot-lda/lmtp with sieve instead of procmail.
>> 
>> Aki
>> 
> Hi Aki, 
> In support of Hajo I've to say that a few days ago I posted a similar issue, 
> and I use dovecot-lda+sieve. My environment has RHEL6 and 7 servers. When I 
> last updated the servers RHEL6 servers mantained 2.2.10-1_14.el6.x86_64 
> version, while RHEL7 updated dovecot from 2.2.10-8.el7.x86_64 to 
> 2.2.36-3.el7.x86_64. When the RHEL7 servers (used for sympa) processed a 
> message for a user, its indexes were corrupted, and the user could't access 
> his inbox through webmail, so I had to delete dovecot.* files from the user 
> mail path to get it working again. 
> My solution was to downgrade dovecot and dovecot-pigeonhole back to 
> 2.2.10-8.el7.x86_64
> Regards
> Gonzalo 
> 

Re: index problems after update

[Gonzalo Palacios Goicolea via dovecot]

El 21/02/2019 a las 10:51, Aki Tuomi via dovecot escribió:

On 21.2.2019 10.53, Hajo Locke via dovecot wrote:

Hello,

Am 20.02.2019 um 10:39 schrieb Aki Tuomi via dovecot:

On 18 February 2019 09:28 Hajo Locke via dovecot
 wrote:


Hello,
     it seems we need a dovecot developers opinion. May be we hit a
bug or cant help ourselves.


Thanks for your answer.

Core dump with backtrace would help, if possible to acquire. Please
refer to https://dovecot.org/bugreport.html for information how to
get a core dump.

Aki


Unfortunately its hard to get a backtrace because dovecot is not
crashing. so it seems to be more a kind of logic problem in code and
no unexpected situation.
yesterday evening i had next incident. I upgraded from 2.2.33.2 to
2.2.36.1, but same behaviour. Also 2.2.36.1 is tricked by the broken
index and delivers no new mails. it starts delivering if i delete
index files. At this point i cant tell if 2.2.36.1 also has same bug
and writes a damaged index, but very likely.  We dont know this
problems with 2.2.22, between 2.2.22 and 2.2.33.2 a change on
mbox-index code must happend which leads to this big problem. So imapd
cant do what he was created for.

For next incident i prepared a 2.3.2.1 on base of Ubuntu 18.10 and
will try this. In my opinion this is a major problem and i expect a
lot of affected people with version > 2.2.22 and classic mbox-storage.

Thanks,
Hajo

We consider mbox + procmail setup somewhat edge case, and if the core
dump does not point into something more generic, it will probably not
get fixed. It is more likely to have this working if you use
dovecot-lda/lmtp with sieve instead of procmail.

Aki


Hi Aki,
In support of Hajo I've to say that a few days ago I posted a similar 
issue, and I use dovecot-lda+sieve. My environment has RHEL6 and 7 
servers. When I last updated the servers RHEL6 servers mantained 
2.2.10-1_14.el6.x86_64 version, while RHEL7 updated dovecot from 
2.2.10-8.el7.x86_64 to 2.2.36-3.el7.x86_64. When the RHEL7 servers (used 
for sympa) processed a message for a user, its indexes were corrupted, 
and the user could't access his inbox through webmail, so I had to 
delete dovecot.* files from the user mail path to get it working again.
My solution was to downgrade dovecot and dovecot-pigeonhole back to 
2.2.10-8.el7.x86_64

Regards

*Gonzalo
*

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

Am 21.02.2019 um 12:18 schrieb Sami Ketola via dovecot:



On 21 Feb 2019, at 12.23, Hajo Locke via dovecot  wrote:
I think mbox+procmail is a classic setup and wide used and good solution for 
many usecases. Same setup we use many years.
We run ~2 mio mailboxes. our automated systems depends on this setup. creating 
mailboxes, managing mailboxes, creating automated filterrules, backupsystem to 
tell something of them. we can not switch our whole mailsetup to work around 
this bug.
How to get a dump if dovecot not crashing but has wrong behaviour? I would like 
to help and provide useful info, but it depends on kind of problem.
I think if a classic setup is not working in dovecot any more, this is a 
serious problem.

In you first email to this thread it says:


Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
service(imap): child 14135 killed with signal 6 (core dumped)
yes, but this is not 2.2.33.2 from Ubuntu 18.04, this happend after 
update to 2.2.36.1
I try to get a coredump from 2.2.36.1, but after my tests the basic 
behaviour is the same as in 2.2.33.2 -> no new mails and no errormessage.

so iam not sure if this is really same problem or different.
I was able now to catch a coredump of 2.2.36.1
I hope this is helpful and leads to the original problem. output of bt 
full is here: https://pastebin.com/awdpN4U3



So imap is crashing and even dumping a core.

Also I must disagree with your mbox+procmail statement. mbox has always been 
very unoptimised mailbox format and everyone should be emphasised not to use it.
Also that combination has always had problems with indexing and file locking. I 
would not use it on high volume mailservers. Or even medium volume mailservers.
Thats true, but worked for long years with dovecot.  mbox+procmail long 
time was only possibility to store mails and iam sure a lot of people 
use it. Was there an official statement that support for mbox+procmail 
was dropped?
Ubuntu 18.04 is fairly new OS and they offer still no 2.3 version (may 
be same bug), they offer 2.2.33.2 and i think this problem will reach 
users just with a delay.

Please help, current situation is very unsatisfying.



Sami



Thanks,
Hajo

Re: index problems after update

[Sami Ketola via dovecot]

> On 21 Feb 2019, at 12.23, Hajo Locke via dovecot  wrote:
> I think mbox+procmail is a classic setup and wide used and good solution for 
> many usecases. Same setup we use many years.
> We run ~2 mio mailboxes. our automated systems depends on this setup. 
> creating mailboxes, managing mailboxes, creating automated filterrules, 
> backupsystem to tell something of them. we can not switch our whole mailsetup 
> to work around this bug.
> How to get a dump if dovecot not crashing but has wrong behaviour? I would 
> like to help and provide useful info, but it depends on kind of problem.
> I think if a classic setup is not working in dovecot any more, this is a 
> serious problem.

In you first email to this thread it says:

> Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
> service(imap): child 14135 killed with signal 6 (core dumped)

So imap is crashing and even dumping a core.

Also I must disagree with your mbox+procmail statement. mbox has always been 
very unoptimised mailbox format and everyone should be emphasised not to use it.
Also that combination has always had problems with indexing and file locking. I 
would not use it on high volume mailservers. Or even medium volume mailservers.

Sami

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

Am 21.02.2019 um 10:51 schrieb Aki Tuomi via dovecot:

On 21.2.2019 10.53, Hajo Locke via dovecot wrote:

Hello,

Am 20.02.2019 um 10:39 schrieb Aki Tuomi via dovecot:

On 18 February 2019 09:28 Hajo Locke via dovecot
 wrote:


Hello,
     it seems we need a dovecot developers opinion. May be we hit a
bug or cant help ourselves.


Thanks for your answer.

Core dump with backtrace would help, if possible to acquire. Please
refer to https://dovecot.org/bugreport.html for information how to
get a core dump.

Aki


Unfortunately its hard to get a backtrace because dovecot is not
crashing. so it seems to be more a kind of logic problem in code and
no unexpected situation.
yesterday evening i had next incident. I upgraded from 2.2.33.2 to
2.2.36.1, but same behaviour. Also 2.2.36.1 is tricked by the broken
index and delivers no new mails. it starts delivering if i delete
index files. At this point i cant tell if 2.2.36.1 also has same bug
and writes a damaged index, but very likely.  We dont know this
problems with 2.2.22, between 2.2.22 and 2.2.33.2 a change on
mbox-index code must happend which leads to this big problem. So imapd
cant do what he was created for.

For next incident i prepared a 2.3.2.1 on base of Ubuntu 18.10 and
will try this. In my opinion this is a major problem and i expect a
lot of affected people with version > 2.2.22 and classic mbox-storage.

Thanks,
Hajo

We consider mbox + procmail setup somewhat edge case, and if the core
dump does not point into something more generic, it will probably not
get fixed. It is more likely to have this working if you use
dovecot-lda/lmtp with sieve instead of procmail.

Aki


I think mbox+procmail is a classic setup and wide used and good solution 
for many usecases. Same setup we use many years.
We run ~2 mio mailboxes. our automated systems depends on this setup. 
creating mailboxes, managing mailboxes, creating automated filterrules, 
backupsystem to tell something of them. we can not switch our whole 
mailsetup to work around this bug.
How to get a dump if dovecot not crashing but has wrong behaviour? I 
would like to help and provide useful info, but it depends on kind of 
problem.
I think if a classic setup is not working in dovecot any more, this is a 
serious problem.


Thanks,
Hajo

Re: index problems after update

[Aki Tuomi via dovecot]

On 21.2.2019 10.53, Hajo Locke via dovecot wrote:
> Hello,
>
> Am 20.02.2019 um 10:39 schrieb Aki Tuomi via dovecot:
>>> On 18 February 2019 09:28 Hajo Locke via dovecot
>>>  wrote:
>>>
>>>
>>> Hello,
>>>     it seems we need a dovecot developers opinion. May be we hit a
>>> bug or cant help ourselves.
>>>    
> Thanks for your answer.
>> Core dump with backtrace would help, if possible to acquire. Please
>> refer to https://dovecot.org/bugreport.html for information how to
>> get a core dump.
>>
>> Aki
>>
> Unfortunately its hard to get a backtrace because dovecot is not
> crashing. so it seems to be more a kind of logic problem in code and
> no unexpected situation.
> yesterday evening i had next incident. I upgraded from 2.2.33.2 to
> 2.2.36.1, but same behaviour. Also 2.2.36.1 is tricked by the broken
> index and delivers no new mails. it starts delivering if i delete
> index files. At this point i cant tell if 2.2.36.1 also has same bug
> and writes a damaged index, but very likely.  We dont know this
> problems with 2.2.22, between 2.2.22 and 2.2.33.2 a change on
> mbox-index code must happend which leads to this big problem. So imapd
> cant do what he was created for.
>
> For next incident i prepared a 2.3.2.1 on base of Ubuntu 18.10 and
> will try this. In my opinion this is a major problem and i expect a
> lot of affected people with version > 2.2.22 and classic mbox-storage.
>
> Thanks,
> Hajo

We consider mbox + procmail setup somewhat edge case, and if the core
dump does not point into something more generic, it will probably not
get fixed. It is more likely to have this working if you use
dovecot-lda/lmtp with sieve instead of procmail.

Aki

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

Am 20.02.2019 um 10:39 schrieb Aki Tuomi via dovecot:

On 18 February 2019 09:28 Hajo Locke via dovecot  wrote:


Hello,
  
  it seems we need a dovecot developers opinion. May be we hit a bug or cant help ourselves.
  
  


Thanks for your answer.

Core dump with backtrace would help, if possible to acquire. Please refer to 
https://dovecot.org/bugreport.html for information how to get a core dump.

Aki

Unfortunately its hard to get a backtrace because dovecot is not 
crashing. so it seems to be more a kind of logic problem in code and no 
unexpected situation.
yesterday evening i had next incident. I upgraded from 2.2.33.2 to 
2.2.36.1, but same behaviour. Also 2.2.36.1 is tricked by the broken 
index and delivers no new mails. it starts delivering if i delete index 
files. At this point i cant tell if 2.2.36.1 also has same bug and 
writes a damaged index, but very likely.  We dont know this problems 
with 2.2.22, between 2.2.22 and 2.2.33.2 a change on mbox-index code 
must happend which leads to this big problem. So imapd cant do what he 
was created for.


For next incident i prepared a 2.3.2.1 on base of Ubuntu 18.10 and will 
try this. In my opinion this is a major problem and i expect a lot of 
affected people with version > 2.2.22 and classic mbox-storage.


Thanks,
Hajo

Re: index problems after update

[Aki Tuomi via dovecot]

> On 18 February 2019 09:28 Hajo Locke via dovecot  wrote:
> 
> 
> Hello,
>  
>  it seems we need a dovecot developers opinion. May be we hit a bug or cant 
> help ourselves.
>  
>  
>

Core dump with backtrace would help, if possible to acquire. Please refer to 
https://dovecot.org/bugreport.html for information how to get a core dump.

Aki

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

thanks for your reply.

Am 15.02.2019 um 17:34 schrieb Michael Slusarz via dovecot:
On February 15, 2019 at 3:35 AM Hajo Locke via dovecot 
 wrote:


Thanks for your reply, so iam not the only one. I think downgrading 
is not an option for me. Ubuntu 18.04 comes with openssl 1.1.0 and it 
seems that older versions like 2.2.22 not compile successful.
Best way for me would be a patch from Ubuntu-side, but without any 
errormessage or something its wasted time to file a bugreport.  
Though Ubuntu 18.04 should run for next 5 years.

https://repo.dovecot.org/

michael
Without statement of dovecot developers i dont like upgrading. I want to 
explain:  One should think that latest versions in 2.2 tree are almost 
bugfree and stable.  But this one is ugly. dovecot crashes its index 
without noticing by itself and hiding new mails. To be particular 
dovecot is not usable in a classic mbox setup with this versions.
Obviously this problem is unknown to developers, else i expected a short 
answer. So it is still thinkable that this problem still exists in 2.3 
versions. At this point it's frustrating and disappointing that this 
case is ignored by developers.  My first mail was 11 days ago.
Not every user is an early adopter with 2.3 versions/mdbox etc. the 
bigger the company the slower upgrades are deployed. We have several 
hundreds of affected servers with 2.2.33.2, we have several thousands 
servers with older versions without this problem, which should get 
updated next weeks/months.
For upgrading i need a version where this problem is fixed (we can 
package by ourselfes if needed). we dont need newest features, just a 
stable classic imapd. we cant use our customers as dummies and test 
different versions to see if a bug is fixed or may be other issues appear.

I hope there is an understanding for our situation.

Thanks,
Hajo

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

it seems we need a dovecot developers opinion. May be we hit a bug or 
cant help ourselves.


Am 15.02.2019 um 11:35 schrieb Hajo Locke via dovecot:

Hello,

Am 15.02.2019 um 10:03 schrieb Gonzalo Palacios Goicolea via dovecot:

El 15/02/2019 a las 8:47, Hajo Locke via dovecot escribió:

Hello,

Am 11.02.2019 um 09:12 schrieb Hajo Locke via dovecot:

Hello,
Am 08.02.2019 um 09:25 schrieb Hajo Locke via dovecot:

Hello List,

i have a problem with index-files which is separated in 2 
subproblems. May be these problems are connected.
Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 
2.2.33.2
These servers are fresh installed machines and users are added to 
the system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There 
is no error in log, dovecot just seems to do his normal operations 
but is not delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap 
folder. So index is recreated and everything works again. I cant 
reproduce this problem, but every 2 days i have one user who is 
reporting this problem.




Iam surprised that no one has an opinion in this case. Nobody else 
noticed this problem?
Yesterday i had a customer who was affected twice. First time we saw 
an errormessage, which leads to an index problem:


Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
Corrupted record in index cache file 
/home/popuser/username/mail/.imap/INBOX/dovecot.index.cache: UID 
26876: Broken physical size in mailbox INBOX: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876)
Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876) (FETCH BODY[] for 
mailbox INBOX UID 26876)


At this point no new incoming mail is delivered by imapd. We deleted 
index files, dovecot reacreated them and all was working again.
a day later same user was affected again. But like all other cases 
before there is no error or unusual logline. i turned mail_debug on, 
but no further hint.
There are definitely new Mails in /var/mail/username but dovecot 
still shows old state from cache. I deleted index-files again and 
imap-client showed a lot of new unread mails.
In log i can see that people are active in imap-clients and managing 
mails. If a new mail is coming in this period it seems to harm 
index-files some times. We never noticed same behaviour with long 
used dovecot 2.2.22


It seems that 2.2.33.2 which is bundled with Ubuntu18.04 LTS has a 
special problem.


Please help. What is your advice?


Second problem is similiar. I Upgraded a server from Ubuntu 16.04 
to Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 
2.2.33.2. After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but 
the difference is we have a Raw backtrace in Log just as expected:



i think this crash is a special problem of dovecot 2.2.36.1.
if i downgrade to 2.2.33.2 all is working well, upgrading again to 
2.2.36.1 leeds to same crash.
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) 
[0x7f78b3b2cee2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) 
[0x7f78b3b2cfda] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) 
[0x7f78b3b38814] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](+0x2081c) [0x5581ac76d81c] -> dovecot/imap [m038422 
ip.ip.ip.ip FETCH](+0x1ea54) [0x5581ac76ba54] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] 
-> dovecot/imap [myuser ip.ip.ip.ip FETCH](cmd_fetch+0x31b) 
[0x5581ac75e1bb] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](command_exec+0x5c) [0x5581ac76a09c] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f78b3b43482] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36)

Re: index problems after update

[Michael Slusarz via dovecot]

> On February 15, 2019 at 3:35 AM Hajo Locke via dovecot  
> wrote:
> 
> Thanks for your reply, so iam not the only one. I think downgrading is 
> not an option for me. Ubuntu 18.04 comes with openssl 1.1.0 and it seems that 
> older versions like 2.2.22 not compile successful.
> Best way for me would be a patch from Ubuntu-side, but without any 
> errormessage or something its wasted time to file a bugreport.  Though Ubuntu 
> 18.04 should run for next 5 years.
> 
https://repo.dovecot.org/

michael

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

Am 15.02.2019 um 10:03 schrieb Gonzalo Palacios Goicolea via dovecot:

El 15/02/2019 a las 8:47, Hajo Locke via dovecot escribió:

Hello,

Am 11.02.2019 um 09:12 schrieb Hajo Locke via dovecot:

Hello,
Am 08.02.2019 um 09:25 schrieb Hajo Locke via dovecot:

Hello List,

i have a problem with index-files which is separated in 2 
subproblems. May be these problems are connected.
Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 
2.2.33.2
These servers are fresh installed machines and users are added to 
the system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There is 
no error in log, dovecot just seems to do his normal operations but 
is not delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap 
folder. So index is recreated and everything works again. I cant 
reproduce this problem, but every 2 days i have one user who is 
reporting this problem.




Iam surprised that no one has an opinion in this case. Nobody else 
noticed this problem?
Yesterday i had a customer who was affected twice. First time we saw 
an errormessage, which leads to an index problem:


Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
Corrupted record in index cache file 
/home/popuser/username/mail/.imap/INBOX/dovecot.index.cache: UID 
26876: Broken physical size in mailbox INBOX: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876)
Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876) (FETCH BODY[] for 
mailbox INBOX UID 26876)


At this point no new incoming mail is delivered by imapd. We deleted 
index files, dovecot reacreated them and all was working again.
a day later same user was affected again. But like all other cases 
before there is no error or unusual logline. i turned mail_debug on, 
but no further hint.
There are definitely new Mails in /var/mail/username but dovecot 
still shows old state from cache. I deleted index-files again and 
imap-client showed a lot of new unread mails.
In log i can see that people are active in imap-clients and managing 
mails. If a new mail is coming in this period it seems to harm 
index-files some times. We never noticed same behaviour with long 
used dovecot 2.2.22


It seems that 2.2.33.2 which is bundled with Ubuntu18.04 LTS has a 
special problem.


Please help. What is your advice?


Second problem is similiar. I Upgraded a server from Ubuntu 16.04 
to Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 
2.2.33.2. After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but the 
difference is we have a Raw backtrace in Log just as expected:



i think this crash is a special problem of dovecot 2.2.36.1.
if i downgrade to 2.2.33.2 all is working well, upgrading again to 
2.2.36.1 leeds to same crash.
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) 
[0x7f78b3b2cee2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) 
[0x7f78b3b2cfda] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) 
[0x7f78b3b38814] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](+0x2081c) [0x5581ac76d81c] -> dovecot/imap [m038422 
ip.ip.ip.ip FETCH](+0x1ea54) [0x5581ac76ba54] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] 
-> dovecot/imap [myuser ip.ip.ip.ip FETCH](cmd_fetch+0x31b) 
[0x5581ac75e1bb] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](command_exec+0x5c) [0x5581ac76a09c] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f78b3b43482] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
[0x7f78b3b43516] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f78b3b436c8] 
-> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7

Re: index problems after update

[Gonzalo Palacios Goicolea via dovecot]

El 15/02/2019 a las 8:47, Hajo Locke via dovecot escribió:

Hello,

Am 11.02.2019 um 09:12 schrieb Hajo Locke via dovecot:

Hello,
Am 08.02.2019 um 09:25 schrieb Hajo Locke via dovecot:

Hello List,

i have a problem with index-files which is separated in 2 
subproblems. May be these problems are connected.
Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 
2.2.33.2
These servers are fresh installed machines and users are added to 
the system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There is 
no error in log, dovecot just seems to do his normal operations but 
is not delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap 
folder. So index is recreated and everything works again. I cant 
reproduce this problem, but every 2 days i have one user who is 
reporting this problem.




Iam surprised that no one has an opinion in this case. Nobody else 
noticed this problem?
Yesterday i had a customer who was affected twice. First time we saw 
an errormessage, which leads to an index problem:


Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
Corrupted record in index cache file 
/home/popuser/username/mail/.imap/INBOX/dovecot.index.cache: UID 
26876: Broken physical size in mailbox INBOX: read(/var/mail/username) 
failed: Cached message size smaller than expected (2118 < 8088, 
box=INBOX, UID=26876)
Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876) (FETCH BODY[] for mailbox 
INBOX UID 26876)


At this point no new incoming mail is delivered by imapd. We deleted 
index files, dovecot reacreated them and all was working again.
a day later same user was affected again. But like all other cases 
before there is no error or unusual logline. i turned mail_debug on, 
but no further hint.
There are definitely new Mails in /var/mail/username but dovecot still 
shows old state from cache. I deleted index-files again and 
imap-client showed a lot of new unread mails.
In log i can see that people are active in imap-clients and managing 
mails. If a new mail is coming in this period it seems to harm 
index-files some times. We never noticed same behaviour with long used 
dovecot 2.2.22


It seems that 2.2.33.2 which is bundled with Ubuntu18.04 LTS has a 
special problem.


Please help. What is your advice?


Second problem is similiar. I Upgraded a server from Ubuntu 16.04 to 
Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 
2.2.33.2. After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but the 
difference is we have a Raw backtrace in Log just as expected:



i think this crash is a special problem of dovecot 2.2.36.1.
if i downgrade to 2.2.33.2 all is working well, upgrading again to 
2.2.36.1 leeds to same crash.
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) 
[0x7f78b3b2cee2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) 
[0x7f78b3b2cfda] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) 
[0x7f78b3b38814] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](+0x2081c) [0x5581ac76d81c] -> dovecot/imap [m038422 
ip.ip.ip.ip FETCH](+0x1ea54) [0x5581ac76ba54] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](cmd_fetch+0x31b) 
[0x5581ac75e1bb] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](command_exec+0x5c) [0x5581ac76a09c] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f78b3b43482] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
[0x7f78b3b43516] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f78b3b436c8] 
-> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f78b3ac6aa3] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](main+0x329) [0x5581ac7

Re: index problems after update

[Hajo Locke via dovecot]

Hello,

Am 11.02.2019 um 09:12 schrieb Hajo Locke via dovecot:

Hello,
Am 08.02.2019 um 09:25 schrieb Hajo Locke via dovecot:

Hello List,

i have a problem with index-files which is separated in 2 
subproblems. May be these problems are connected.

Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 2.2.33.2
These servers are fresh installed machines and users are added to the 
system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There is 
no error in log, dovecot just seems to do his normal operations but 
is not delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap folder. 
So index is recreated and everything works again. I cant reproduce 
this problem, but every 2 days i have one user who is reporting this 
problem.




Iam surprised that no one has an opinion in this case. Nobody else 
noticed this problem?
Yesterday i had a customer who was affected twice. First time we saw an 
errormessage, which leads to an index problem:


Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
Corrupted record in index cache file 
/home/popuser/username/mail/.imap/INBOX/dovecot.index.cache: UID 26876: 
Broken physical size in mailbox INBOX: read(/var/mail/username) failed: 
Cached message size smaller than expected (2118 < 8088, box=INBOX, 
UID=26876)
Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876) (FETCH BODY[] for mailbox 
INBOX UID 26876)


At this point no new incoming mail is delivered by imapd. We deleted 
index files, dovecot reacreated them and all was working again.
a day later same user was affected again. But like all other cases 
before there is no error or unusual logline. i turned mail_debug on, but 
no further hint.
There are definitely new Mails in /var/mail/username but dovecot still 
shows old state from cache. I deleted index-files again and imap-client 
showed a lot of new unread mails.
In log i can see that people are active in imap-clients and managing 
mails. If a new mail is coming in this period it seems to harm 
index-files some times. We never noticed same behaviour with long used 
dovecot 2.2.22


It seems that 2.2.33.2 which is bundled with Ubuntu18.04 LTS has a 
special problem.


Please help. What is your advice?


Second problem is similiar. I Upgraded a server from Ubuntu 16.04 to 
Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 
2.2.33.2. After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but the 
difference is we have a Raw backtrace in Log just as expected:



i think this crash is a special problem of dovecot 2.2.36.1.
if i downgrade to 2.2.33.2 all is working well, upgrading again to 
2.2.36.1 leeds to same crash.
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) 
[0x7f78b3b2cee2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) 
[0x7f78b3b2cfda] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) [0x7f78b3b38814] 
-> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x2081c) 
[0x5581ac76d81c] -> dovecot/imap [m038422 ip.ip.ip.ip 
FETCH](+0x1ea54) [0x5581ac76ba54] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](cmd_fetch+0x31b) [0x5581ac75e1bb] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](command_exec+0x5c) [0x5581ac76a09c] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f78b3b43482] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
[0x7f78b3b43516] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f78b3b436c8] 
-> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f78b3ac6aa3] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](main+0x329) [0x5581ac75b159] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0

Re: index problems after update

[Hajo Locke via dovecot]

Hello,
Am 08.02.2019 um 09:25 schrieb Hajo Locke via dovecot:

Hello List,

i have a problem with index-files which is separated in 2 subproblems. 
May be these problems are connected.

Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 2.2.33.2
These servers are fresh installed machines and users are added to the 
system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There is no 
error in log, dovecot just seems to do his normal operations but is 
not delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap folder. 
So index is recreated and everything works again. I cant reproduce 
this problem, but every 2 days i have one user who is reporting this 
problem.


Second problem is similiar. I Upgraded a server from Ubuntu 16.04 to 
Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 
2.2.33.2. After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but the 
difference is we have a Raw backtrace in Log just as expected:



i think this crash is a special problem of dovecot 2.2.36.1.
if i downgrade to 2.2.33.2 all is working well, upgrading again to 
2.2.36.1 leeds to same crash.
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) [0x7f78b3b2cee2] 
-> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) [0x7f78b3b2cfda] -> 
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) [0x7f78b3b38814] 
-> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x2081c) 
[0x5581ac76d81c] -> dovecot/imap [m038422 ip.ip.ip.ip FETCH](+0x1ea54) 
[0x5581ac76ba54] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](cmd_fetch+0x31b) [0x5581ac75e1bb] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](command_exec+0x5c) [0x5581ac76a09c] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f78b3b43482] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
[0x7f78b3b43516] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f78b3b436c8] -> 
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f78b3ac6aa3] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](main+0x329) [0x5581ac75b159] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
[0x7f78b36bbb97] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](_start+0x2a) [0x5581ac75b2fa]
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
service(imap): child 14135 killed with signal 6 (core dumped)


Over all dovecot and OS versions we use same old-fashioned setup. 
Mails are delivered by procmail to classic mbox file.

This is output of doveconf -n

# 2.2.36.1 (5d621cf65): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
# OS: Linux 4.18.0-14-generic x86_64 Ubuntu 18.04.1 LTS
# Hostname: myhostname
auth_mechanisms = plain login
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@üÜöÖäÄ

auth_worker_max_count = 60
disable_plaintext_auth = no
imap_capability = +XLIST
lock_method = flock
login_greeting = mailserver ready.
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
mail_cache_min_mail_count = 30
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_max_userip_connections = 30
mbox_read_locks = dotlock
mbox_very_dirty_syncs = yes
namespace inbox {
  inbox = yes
  location =
  mailbox Archiv {
    auto = no
    special_use =
  }
  mailbox Drafts {
    auto = no
    special_use =
  }
  mailbox Junk {
    auto = no
    special_use =
  }
  mailbox Sent {
    auto = no
    special_use =
  }
  mailbox Trash {
    auto = no
    special_use =
  }
  prefix =
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
protocols = imap pop3
service auth-worker {
  idle_kill = 1 mins
}
service auth {
  idle_kill = 1 mins
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
 

index problems after update

[Hajo Locke via dovecot]

Hello List,

i have a problem with index-files which is separated in 2 subproblems. 
May be these problems are connected.

Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 2.2.33.2
These servers are fresh installed machines and users are added to the 
system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There is no 
error in log, dovecot just seems to do his normal operations but is not 
delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap folder. So 
index is recreated and everything works again. I cant reproduce this 
problem, but every 2 days i have one user who is reporting this problem.


Second problem is similiar. I Upgraded a server from Ubuntu 16.04 to 
Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 2.2.33.2. 
After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but the 
difference is we have a Raw backtrace in Log just as expected:


Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) [0x7f78b3b2cee2] 
-> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) [0x7f78b3b2cfda] -> 
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) 
[0x7f78b3b38814] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x2081c) 
[0x5581ac76d81c] -> dovecot/imap [m038422 ip.ip.ip.ip FETCH](+0x1ea54) 
[0x5581ac76ba54] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](cmd_fetch+0x31b) [0x5581ac75e1bb] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](command_exec+0x5c) [0x5581ac76a09c] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f78b3b43482] 
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
[0x7f78b3b43516] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f78b3b436c8] -> 
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f78b3ac6aa3] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](main+0x329) 
[0x5581ac75b159] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f78b36bbb97] 
-> dovecot/imap [myuser ip.ip.ip.ip FETCH](_start+0x2a) [0x5581ac75b2fa]
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
service(imap): child 14135 killed with signal 6 (core dumped)


Over all dovecot and OS versions we use same old-fashioned setup. Mails 
are delivered by procmail to classic mbox file.

This is output of doveconf -n

# 2.2.36.1 (5d621cf65): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
# OS: Linux 4.18.0-14-generic x86_64 Ubuntu 18.04.1 LTS
# Hostname: myhostname
auth_mechanisms = plain login
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@üÜöÖäÄ

auth_worker_max_count = 60
disable_plaintext_auth = no
imap_capability = +XLIST
lock_method = flock
login_greeting = mailserver ready.
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
mail_cache_min_mail_count = 30
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_max_userip_connections = 30
mbox_read_locks = dotlock
mbox_very_dirty_syncs = yes
namespace inbox {
  inbox = yes
  location =
  mailbox Archiv {
    auto = no
    special_use =
  }
  mailbox Drafts {
    auto = no
    special_use =
  }
  mailbox Junk {
    auto = no
    special_use =
  }
  mailbox Sent {
    auto = no
    special_use =
  }
  mailbox Trash {
    auto = no
    special_use =
  }
  prefix =
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
protocols = imap pop3
service auth-worker {
  idle_kill = 1 mins
}
service auth {
  idle_kill = 1 mins
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
  user = root
}
service imap-login {
  executable = /usr/lib/dovecot/imap-login
  process_min_avail = 4
  service_count = 0
  user = dovecot
}
service imap {
  process_limit = 1024
}
service pop3-login {
  executable = /usr