Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-28 Thread Timo Sirainen
On Thu, 2010-02-25 at 21:04 +0200, Timo Sirainen wrote:
> Looks like some input stream seeking optimizations are broken (when
> one input stream reads from another, which reads from another, ...). I
> already managed to fix the performance problem, but now it's
> corrupting saved mails sometimes. So a while longer to get it fully
> fixed :) And since it's a pretty big change, I'm not sure if I want to
> risk breaking v1.2 by changing it, so maybe it's v2.0 only.

Fixed now in v2.0 and optimized also conversion to uppercase:

http://hg.dovecot.org/dovecot-2.0/rev/de2798fbbae6
http://hg.dovecot.org/dovecot-2.0/rev/23858ce6422e

After this, searching your spam mailbox goes from:

v1.2 OK Search completed (26.084 secs).

to

v2.0 OK Search completed (2.172 secs).

It's still slower than v1.0's 1.83 secs, but the increased slowness is
most likely because v1.0 didn't support case-insensitive unicode
searches.



signature.asc
Description: This is a digitally signed message part


Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-25 Thread Timo Sirainen
On 24.2.2010, at 20.27, Timo Sirainen wrote:

> Looks like there's something very wrong with mbox with v1.2+. It's doing
> a *lot* of message header parsing work that doesn't happen with v1.1 or
> with other mailbox formats. Probably because I fixed some bugs where it
> was wrongly caching some state, but now it's not caching it long enough.

Looks like some input stream seeking optimizations are broken (when one input 
stream reads from another, which reads from another, ...). I already managed to 
fix the performance problem, but now it's corrupting saved mails sometimes. So 
a while longer to get it fully fixed :) And since it's a pretty big change, I'm 
not sure if I want to risk breaking v1.2 by changing it, so maybe it's v2.0 
only.



Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-24 Thread Eric Rostetter

Quoting Stan Hoeppner :


I'm running mbox with 1.2 and not seeing any problems...  But that may be
because I threw a lot of hardware at it?


Hi Eric.  Not sure if even fast hardware searching 11,000+ message count
mbox'en without an FTS plugin would give speedy results, given Timo's
discovery of earlier today.


Can't say, as I don't personally have a 11K+ message mbox handy that I
can run the test on.  I just don't keep that much mail around of my own...

But it works okay on my 4K to 5K message mbox files, which are the largest
I have...  Usually takes about 1 second per 1K messages, so about 4 seconds
for the 4K mbox, 5 seconds for the 5K mbox, etc.  Of course, a bit slower
when the server is overly busy...  I've had it take 10 or 12 seconds before...

I do have users with large mbox files (12K+) who have never complained,
but that doesn't mean much...  I don't know which ones do body searches,
or how often they do them, etc.  And not all users complain even when there
is a slowness or outright problem...

So, no scientific data... But no complaints here...  Then again, I only have
a small number of users with mboxes that large... We discourage that kind of
thing here...

--
Eric Rostetter
The Department of Physics
The University of Texas at Austin

Go Longhorns!


Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-24 Thread Stan Hoeppner
Eric Rostetter put forth on 2/24/2010 9:13 PM:
> Quoting Stan Hoeppner :
> 
>> I guess you didn't have enough (any?) people testing mbox body search
>> before
>> the v1.2 release.  Is everyone but me using maildir?  Makes me wish I
>> had an
>> extra box so I could do dovecot devel version testing against mbox.
> 
> I'm running mbox with 1.2 and not seeing any problems...  But that may be
> because I threw a lot of hardware at it?

Hi Eric.  Not sure if even fast hardware searching 11,000+ message count
mbox'en without an FTS plugin would give speedy results, given Timo's
discovery of earlier today.

Timo Sirainen put forth on 2/24/2010 12:27 PM:
> Looks like there's something very wrong with mbox with v1.2+. It's doing
> a *lot* of message header parsing work that doesn't happen with v1.1 or
> with other mailbox formats. Probably because I fixed some bugs where it
> was wrongly caching some state, but now it's not caching it long enough.

-- 
Stan


Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-24 Thread Eric Rostetter

Quoting Stan Hoeppner :


I guess you didn't have enough (any?) people testing mbox body search before
the v1.2 release.  Is everyone but me using maildir?  Makes me wish I had an
extra box so I could do dovecot devel version testing against mbox.


I'm running mbox with 1.2 and not seeing any problems...  But that may be
because I threw a lot of hardware at it?


--
Stan


--
Eric Rostetter
The Department of Physics
The University of Texas at Austin

Go Longhorns!


Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-24 Thread Stan Hoeppner
Timo Sirainen put forth on 2/24/2010 12:27 PM:

> Well, when index is up-to-date it's fast. But after you've received a
> few mails, at least with me it seemed to spend more time updating the
> index than just doing the regular search.

Ahh, I see what you mean.  It's slower after new mail arrives, 20 seconds or
so, but it's still much much faster than without squat which was multiple
minutes.

> Looks like there's something very wrong with mbox with v1.2+. It's doing
> a *lot* of message header parsing work that doesn't happen with v1.1 or
> with other mailbox formats. Probably because I fixed some bugs where it
> was wrongly caching some state, but now it's not caching it long enough.

I guess you didn't have enough (any?) people testing mbox body search before
the v1.2 release.  Is everyone but me using maildir?  Makes me wish I had an
extra box so I could do dovecot devel version testing against mbox.

-- 
Stan


Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-24 Thread Timo Sirainen
On Wed, 2010-02-24 at 01:21 -0600, Stan Hoeppner wrote:
> Stan Hoeppner put forth on 2/24/2010 12:17 AM:
> > Timo Sirainen put forth on 2/23/2010 11:06 PM:
> 
> >> No, body searches aren't indexed (without fts plugins).
> 
> I wish you'd have mentioned the Squat plugin earlier Timo.  I feel I've made
> a fool of myself.  I just setup squat and after a long first search on that
> 50MB, 11K+ messages mbox file, every search after the first has taken less
> than 5 seconds, most only 2 seconds.  It's performing exactly as I'd like.
> It's amazing.

Well, when index is up-to-date it's fast. But after you've received a
few mails, at least with me it seemed to spend more time updating the
index than just doing the regular search.

> I didn't ever setup any fts plugins on 1.0.15.  I'm left wondering how body
> searches were so fast if I didn't have an fts plugin configured.

Looks like there's something very wrong with mbox with v1.2+. It's doing
a *lot* of message header parsing work that doesn't happen with v1.1 or
with other mailbox formats. Probably because I fixed some bugs where it
was wrongly caching some state, but now it's not caching it long enough.


signature.asc
Description: This is a digitally signed message part


Re: [Dovecot] **OFF LIST** Re: body search very slow since upgrade from 1.0.15 to 1.2.10

2010-02-23 Thread Stan Hoeppner
Stan Hoeppner put forth on 2/24/2010 12:17 AM:
> Timo Sirainen put forth on 2/23/2010 11:06 PM:

>> No, body searches aren't indexed (without fts plugins).

I wish you'd have mentioned the Squat plugin earlier Timo.  I feel I've made
a fool of myself.  I just setup squat and after a long first search on that
50MB, 11K+ messages mbox file, every search after the first has taken less
than 5 seconds, most only 2 seconds.  It's performing exactly as I'd like.
It's amazing.

I didn't ever setup any fts plugins on 1.0.15.  I'm left wondering how body
searches were so fast if I didn't have an fts plugin configured.

Anyway, setting up Squat on 1.2.10 sure solved the long search time problem. :)

Thanks for the help Timo!

-- 
Stan