Bug#518093: Very slow, high CPU when syncing after thousands of remote deletions.

2009-03-04 Thread John Goerzen
Trent W. Buck wrote:
> Package: offlineimap
> Version: 6.0.3
> Severity: normal
> 
> About once a year, I connect to imap.gmail.com with mutt and delete
> lots of mail -- duplicates, anything archived on a mailing list, spam,
> etc.
> 
> I did this just now, and ended up deleting around 20,000 messages.
> When I run offlineimap, it happily prints that it will delete stuff
> from "[...]) in Maildir[all], LocalStatus[all]".  And indeed, almost
> immediately I can see with du that the maildir folder has gone from
> 290MB to 100MB.  Admirably efficient.
> 
> But offlineimap doesn't stop.  So far it has been running for
> twenty-five minutes, and in that time has used up 13 minutes of CPU
> (i.e. about 50% CPU utilization).
> 
> Clearly something extremely suboptimal is going on here.  If I had to
> guess, I would suspect the problem is that offlineimap is repeatedly
> opening its ~/.offlineimap metadata, deleting a single message, then
> closing those files again.
> 
> This problem may also be exacerbated by the fact that my nonvolatile
> storage is an SSD, which (I hear) has slow write speeds compared to a
> traditional HDD.  The disk activity light is blinking at around 2Hz.

OfflineIMAP flushes the cache after each write.  This will likely be
extremely slow on SSD.  It's known to be slow in general.  Some have
proposed SQLite for the cache, but the patches posted to date haven't
been suitable for integration into OfflineIMAP yet.

In the meantime, you can experiment with setting doautosave=0 in
offlineimap/folder/LocalStatus.py.  However, this hasn't been tested, so
I have no idea if it will dramatically improve performance or wipe out
all your mail.

-- John

> 
> -- System Information:
> Debian Release: 5.0
>   APT prefers testing
>   APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (1, 
> 'experimental')
> Architecture: i386 (i686)
> 
> Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core)
> Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> 
> Versions of packages offlineimap depends on:
> ii  python2.5.4-2An interactive high-level 
> object-o
> ii  python-support0.8.7  automated rebuilding support for 
> P
> 
> offlineimap recommends no packages.
> 
> Versions of packages offlineimap suggests:
> pn  python-kerberos(no description available)
> 
> -- no debconf information
> 
> 
> 




-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org



Bug#518093: Very slow, high CPU when syncing after thousands of remote deletions.

2009-03-03 Thread Trent W. Buck
Package: offlineimap
Version: 6.0.3
Severity: normal

About once a year, I connect to imap.gmail.com with mutt and delete
lots of mail -- duplicates, anything archived on a mailing list, spam,
etc.

I did this just now, and ended up deleting around 20,000 messages.
When I run offlineimap, it happily prints that it will delete stuff
from "[...]) in Maildir[all], LocalStatus[all]".  And indeed, almost
immediately I can see with du that the maildir folder has gone from
290MB to 100MB.  Admirably efficient.

But offlineimap doesn't stop.  So far it has been running for
twenty-five minutes, and in that time has used up 13 minutes of CPU
(i.e. about 50% CPU utilization).

Clearly something extremely suboptimal is going on here.  If I had to
guess, I would suspect the problem is that offlineimap is repeatedly
opening its ~/.offlineimap metadata, deleting a single message, then
closing those files again.

This problem may also be exacerbated by the fact that my nonvolatile
storage is an SSD, which (I hear) has slow write speeds compared to a
traditional HDD.  The disk activity light is blinking at around 2Hz.

-- System Information:
Debian Release: 5.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (1, 
'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages offlineimap depends on:
ii  python2.5.4-2An interactive high-level object-o
ii  python-support0.8.7  automated rebuilding support for P

offlineimap recommends no packages.

Versions of packages offlineimap suggests:
pn  python-kerberos(no description available)

-- no debconf information



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org