Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
On Thu, 2010-02-25 at 21:04 +0200, Timo Sirainen wrote: > Looks like some input stream seeking optimizations are broken (when > one input stream reads from another, which reads from another, ...). I > already managed to fix the performance problem, but now it's > corrupting saved mails sometimes. So a while longer to get it fully > fixed :) And since it's a pretty big change, I'm not sure if I want to > risk breaking v1.2 by changing it, so maybe it's v2.0 only. Fixed now in v2.0 and optimized also conversion to uppercase: http://hg.dovecot.org/dovecot-2.0/rev/de2798fbbae6 http://hg.dovecot.org/dovecot-2.0/rev/23858ce6422e After this, searching your spam mailbox goes from: v1.2 OK Search completed (26.084 secs). to v2.0 OK Search completed (2.172 secs). It's still slower than v1.0's 1.83 secs, but the increased slowness is most likely because v1.0 didn't support case-insensitive unicode searches. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
On 24.2.2010, at 20.27, Timo Sirainen wrote: > Looks like there's something very wrong with mbox with v1.2+. It's doing > a *lot* of message header parsing work that doesn't happen with v1.1 or > with other mailbox formats. Probably because I fixed some bugs where it > was wrongly caching some state, but now it's not caching it long enough. Looks like some input stream seeking optimizations are broken (when one input stream reads from another, which reads from another, ...). I already managed to fix the performance problem, but now it's corrupting saved mails sometimes. So a while longer to get it fully fixed :) And since it's a pretty big change, I'm not sure if I want to risk breaking v1.2 by changing it, so maybe it's v2.0 only.
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
Quoting Stan Hoeppner : I'm running mbox with 1.2 and not seeing any problems... But that may be because I threw a lot of hardware at it? Hi Eric. Not sure if even fast hardware searching 11,000+ message count mbox'en without an FTS plugin would give speedy results, given Timo's discovery of earlier today. Can't say, as I don't personally have a 11K+ message mbox handy that I can run the test on. I just don't keep that much mail around of my own... But it works okay on my 4K to 5K message mbox files, which are the largest I have... Usually takes about 1 second per 1K messages, so about 4 seconds for the 4K mbox, 5 seconds for the 5K mbox, etc. Of course, a bit slower when the server is overly busy... I've had it take 10 or 12 seconds before... I do have users with large mbox files (12K+) who have never complained, but that doesn't mean much... I don't know which ones do body searches, or how often they do them, etc. And not all users complain even when there is a slowness or outright problem... So, no scientific data... But no complaints here... Then again, I only have a small number of users with mboxes that large... We discourage that kind of thing here... -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns!
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
Eric Rostetter put forth on 2/24/2010 9:13 PM: > Quoting Stan Hoeppner : > >> I guess you didn't have enough (any?) people testing mbox body search >> before >> the v1.2 release. Is everyone but me using maildir? Makes me wish I >> had an >> extra box so I could do dovecot devel version testing against mbox. > > I'm running mbox with 1.2 and not seeing any problems... But that may be > because I threw a lot of hardware at it? Hi Eric. Not sure if even fast hardware searching 11,000+ message count mbox'en without an FTS plugin would give speedy results, given Timo's discovery of earlier today. Timo Sirainen put forth on 2/24/2010 12:27 PM: > Looks like there's something very wrong with mbox with v1.2+. It's doing > a *lot* of message header parsing work that doesn't happen with v1.1 or > with other mailbox formats. Probably because I fixed some bugs where it > was wrongly caching some state, but now it's not caching it long enough. -- Stan
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
Quoting Stan Hoeppner : I guess you didn't have enough (any?) people testing mbox body search before the v1.2 release. Is everyone but me using maildir? Makes me wish I had an extra box so I could do dovecot devel version testing against mbox. I'm running mbox with 1.2 and not seeing any problems... But that may be because I threw a lot of hardware at it? -- Stan -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns!
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
Timo Sirainen put forth on 2/24/2010 12:27 PM: > Well, when index is up-to-date it's fast. But after you've received a > few mails, at least with me it seemed to spend more time updating the > index than just doing the regular search. Ahh, I see what you mean. It's slower after new mail arrives, 20 seconds or so, but it's still much much faster than without squat which was multiple minutes. > Looks like there's something very wrong with mbox with v1.2+. It's doing > a *lot* of message header parsing work that doesn't happen with v1.1 or > with other mailbox formats. Probably because I fixed some bugs where it > was wrongly caching some state, but now it's not caching it long enough. I guess you didn't have enough (any?) people testing mbox body search before the v1.2 release. Is everyone but me using maildir? Makes me wish I had an extra box so I could do dovecot devel version testing against mbox. -- Stan
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
On Wed, 2010-02-24 at 01:21 -0600, Stan Hoeppner wrote: > Stan Hoeppner put forth on 2/24/2010 12:17 AM: > > Timo Sirainen put forth on 2/23/2010 11:06 PM: > > >> No, body searches aren't indexed (without fts plugins). > > I wish you'd have mentioned the Squat plugin earlier Timo. I feel I've made > a fool of myself. I just setup squat and after a long first search on that > 50MB, 11K+ messages mbox file, every search after the first has taken less > than 5 seconds, most only 2 seconds. It's performing exactly as I'd like. > It's amazing. Well, when index is up-to-date it's fast. But after you've received a few mails, at least with me it seemed to spend more time updating the index than just doing the regular search. > I didn't ever setup any fts plugins on 1.0.15. I'm left wondering how body > searches were so fast if I didn't have an fts plugin configured. Looks like there's something very wrong with mbox with v1.2+. It's doing a *lot* of message header parsing work that doesn't happen with v1.1 or with other mailbox formats. Probably because I fixed some bugs where it was wrongly caching some state, but now it's not caching it long enough. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10
Stan Hoeppner put forth on 2/24/2010 12:17 AM: > Timo Sirainen put forth on 2/23/2010 11:06 PM: >> No, body searches aren't indexed (without fts plugins). I wish you'd have mentioned the Squat plugin earlier Timo. I feel I've made a fool of myself. I just setup squat and after a long first search on that 50MB, 11K+ messages mbox file, every search after the first has taken less than 5 seconds, most only 2 seconds. It's performing exactly as I'd like. It's amazing. I didn't ever setup any fts plugins on 1.0.15. I'm left wondering how body searches were so fast if I didn't have an fts plugin configured. Anyway, setting up Squat on 1.2.10 sure solved the long search time problem. :) Thanks for the help Timo! -- Stan
