Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9

[Wolfgang Breyha via Info-cyrus]

On 18/11/16 01:07, Bron Gondwana via Info-cyrus wrote:
> On Fri, 18 Nov 2016, at 10:51, Wolfgang Breyha via Info-cyrus wrote:
>> I already filed a bug https://github.com/cyrusimap/cyrus-imapd/issues/43 
>> but no response so far. I directly asked Bron, but no response as well.
> 
> Sorry, I really don't have a clue.  2.5 does have a different mailboxes.db
> format, so it's a bit more CPU intensive.  The real massive win for CPU
> usage is going to come with reverse ACLs:

Thanks for the response in the first place! I'm sorry to push this topic
continuously because I appreciate all your hard work (and Ellies as well) on
cyrus very much!

We had a quite hard time keeping our infrastructure operational after doing
the final step upgrading our frontends to 2.5. Seeing others having the same
stress should be quite a warning for people thinking about an upgrade.

Since Deniss has these troubles with skiplist as well on 2.5 it looks like the
size
triggers it. twoskip seems to perform even worse if mailboxes.db reaches a
certain size,
or it's only the size penalty which hits twoskip earlier.
150-200 MB seems critical for locking issues.

> https://blog.fastmail.com/2015/12/05/reverse-acls-making-imap-list-fast/

I read about them some time ago in the 3.0.0-beta changelog. Hopefully we're
on a stable cyrus release supporting it before reaching the critical size for
skiplist;-)

> But to get there, we need to solve reverse ACLs for groups.  I did ask about 
> it here:
> 
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-November/038628.html

Sorry, we don't use groups.

> But then didn't follow up to add group reverse ACL support in Cyrus, so 
> reverse ACLs are broken if you're using groups.

So, 2.5 wont get them anyway I guess;-)

Greetings, Wolfgang
-- 
Wolfgang Breyha  | http://www.blafasel.at/
Vienna University Computer Center | Austria


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9

[Bron Gondwana via Info-cyrus]

On Fri, 18 Nov 2016, at 10:51, Wolfgang Breyha via Info-cyrus wrote:
> On 17/11/16 14:00, Deniss via Info-cyrus wrote:
> > Any ideas or suggestion for investigation ?
> 
> I already filed a bug
> https://github.com/cyrusimap/cyrus-imapd/issues/43
> but no response so far. I directly asked Bron, but no response as well.

Sorry, I really don't have a clue.  2.5 does have a different mailboxes.db 
format, so it's a bit more CPU intensive.  The real massive win for CPU usage 
is going to come with reverse ACLs:

https://blog.fastmail.com/2015/12/05/reverse-acls-making-imap-list-fast/

But to get there, we need to solve reverse ACLs for groups.  I did ask about it 
here:

https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-November/038628.html

But then didn't follow up to add group reverse ACL support in Cyrus, so reverse 
ACLs are broken if you're using groups.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9

[Wolfgang Breyha via Info-cyrus]

On 17/11/16 14:00, Deniss via Info-cyrus wrote:
> Hello,
> 
> I trying to migrate one big cyrus imap server from 2.4 to 2.5.9.
> 
> I updated binaries, fix db backend in imapd.conf and converted
> mailboxes.db with ctl_mboxlist -d & -u to twoskip.
> 
> cyrus ran fine until morning when a count of simultanious sessions
> started to rise.

Sounds familiar;-)

> I converted mailboxes.db back to skiplist and got a bunch (10-20 per
> sec) of following logs:

I did the same. twoskip was and is useless on my frontends. With skiplist my
loads dropped at least down to 2-5. After doubling our vCPUs it runs quite well.

Do you use roundcube by any chance? If yes update to 2.5.10 and do not use
$config['imap_disabled_caps'] = array('LIST-EXTENDED');

Otherwise roundcube sends two 'LIST "" "*"' on each login and triggers two
full table scans on your mailboxes.db

> 180M /var/imap/mailboxes.db.skiplist.24
> 215M /var/imap/mailboxes.db.skiplist.25
> 262M /var/imap/mailboxes.db.twoskip.25

Wow, larger then ours. We've ~140MB on skiplist.25.

> Any ideas or suggestion for investigation ?

I already filed a bug
https://github.com/cyrusimap/cyrus-imapd/issues/43
but no response so far. I directly asked Bron, but no response as well.

Greetings, Wolfgang
-- 
Wolfgang Breyha  | http://www.blafasel.at/
Vienna University Computer Center | Austria


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9

[Shawn Bakhtiar via Info-cyrus]

Did you run reconstruct like the upgrade documentation recommends?

When I did my upgrade, from 2.4.x to 2.5 that's about all I had to do. I don't 
recall having to run ctl_mboxlist -du

https://www.cyrusimap.org/docs/cyrus-imapd/2.5.0/install-upgrade.php

http://www.cyrusimap.org/~vanmeeuwen/imap/release-notes/2.5.0.html


On Nov 17, 2016, at 5:00 AM, Deniss via Info-cyrus 
> wrote:

Hello,

I trying to migrate one big cyrus imap server from 2.4 to 2.5.9.

I updated binaries, fix db backend in imapd.conf and converted
mailboxes.db with ctl_mboxlist -d & -u to twoskip.

cyrus ran fine until morning when a count of simultanious sessions
started to rise.

then imapd processes become locked on mailboxes.db when logs show
something like
Nov 17 08:04:21 srv1 imap[11019]: cmdtimer: 
'y...@mail.xxx' 'list'
'' '46.741322' '0.00' '46.741322'
Nov 17 08:04:21 srv1 imap[11019]: buf: A1 LIST "" "*"

I converted mailboxes.db back to skiplist and got a bunch (10-20 per
sec) of following logs:

Nov 17 09:59:09 srv1 imap[5352]: skiplist: longlock
/var/imap/mailboxes.db for 177.7 seconds
Nov 17 09:59:09 srv1 imap[6760]: skiplist: longlock
/var/imap/mailboxes.db for 169.7 seconds
Nov 17 09:59:09 srv1 imap[6764]: skiplist: longlock
/var/imap/mailboxes.db for 171.7 seconds
Nov 17 09:59:09 srv1 imap[5230]: skiplist: longlock
/var/imap/mailboxes.db for 171.8 seconds
Nov 17 09:59:09 srv1 imap[5251]: skiplist: longlock
/var/imap/mailboxes.db for 203.2 seconds
Nov 17 09:59:09 srv1 imap[5221]: skiplist: longlock
/var/imap/mailboxes.db for 168.9 seconds


then I rolled back to cyrus 2.4 (converting mailboxes.db back to old
format) and the server started to work just fine again.

I run 8core 16gb box with mailboxes.db on SSD disk, noswap,
Linux 4.4.8-hardened glibc-2.20 x86_64-pc-linux-gnu-4.8.4 on Gentoo

mailboxes.db is ~200Mb depending of format
135M /var/imap/mailboxes.plain
180M /var/imap/mailboxes.db.skiplist.24
215M /var/imap/mailboxes.db.skiplist.25
262M /var/imap/mailboxes.db.twoskip.25

Any ideas or suggestion for investigation ?

Best,
Deniss

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

mailboxes.db locking problem after updating from 2.4 to 2.5.9

[Deniss via Info-cyrus]

Hello,

I trying to migrate one big cyrus imap server from 2.4 to 2.5.9.

I updated binaries, fix db backend in imapd.conf and converted
mailboxes.db with ctl_mboxlist -d & -u to twoskip.

cyrus ran fine until morning when a count of simultanious sessions
started to rise.

then imapd processes become locked on mailboxes.db when logs show
something like
Nov 17 08:04:21 srv1 imap[11019]: cmdtimer: 'y...@mail.xxx' 'list'
'' '46.741322' '0.00' '46.741322'
Nov 17 08:04:21 srv1 imap[11019]: buf: A1 LIST "" "*"

I converted mailboxes.db back to skiplist and got a bunch (10-20 per
sec) of following logs:

Nov 17 09:59:09 srv1 imap[5352]: skiplist: longlock
/var/imap/mailboxes.db for 177.7 seconds
Nov 17 09:59:09 srv1 imap[6760]: skiplist: longlock
/var/imap/mailboxes.db for 169.7 seconds
Nov 17 09:59:09 srv1 imap[6764]: skiplist: longlock
/var/imap/mailboxes.db for 171.7 seconds
Nov 17 09:59:09 srv1 imap[5230]: skiplist: longlock
/var/imap/mailboxes.db for 171.8 seconds
Nov 17 09:59:09 srv1 imap[5251]: skiplist: longlock
/var/imap/mailboxes.db for 203.2 seconds
Nov 17 09:59:09 srv1 imap[5221]: skiplist: longlock
/var/imap/mailboxes.db for 168.9 seconds


then I rolled back to cyrus 2.4 (converting mailboxes.db back to old
format) and the server started to work just fine again.

I run 8core 16gb box with mailboxes.db on SSD disk, noswap,
Linux 4.4.8-hardened glibc-2.20 x86_64-pc-linux-gnu-4.8.4 on Gentoo

mailboxes.db is ~200Mb depending of format
135M /var/imap/mailboxes.plain
180M /var/imap/mailboxes.db.skiplist.24
215M /var/imap/mailboxes.db.skiplist.25
262M /var/imap/mailboxes.db.twoskip.25

Any ideas or suggestion for investigation ?

Best,
Deniss

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus