Re: offlineimap much slower than gmail-imap
On Mon, 02 May 2011 at 20:52:29 +0200, Toby Cubitt wrote: On Mon, May 02, 2011 at 02:17:26PM +0200, Jose M Vidal wrote: No way: just set the caching just for headers, but the response is still very slow. Any other clue I can follow? I use offlineimap with mutt, and found that switching to a maildir containing of the order of 10,000 emails was a somewhat slow (taking perhaps a minute or so), even with maildir header caching configured. I recompiled mutt with tokyo-cabinet support (which allows mutt to use a tokyo cabinet database for the backend header-cache database), and opening the same maildir now takes a couple of seconds at most. For reference, mutt (1.5.20) on my machine currently takes 11 seconds to open my offlineimap'd Gmail 'All Mail' folder which contains 17,418 messages. It then takes a further 6 seconds to close the mailbox, write any changes, and then switch back to my Inbox (for example). This is on an Apple MacBook Air with SSD, and mutt has been configured with header-cacheing enabled and support for tokyo-cabinet compiled therein. For me this is still too slow, but I'm not sure that there's much else that can be done to improve it. -- -Nick
Re: offlineimap much slower than gmail-imap
On 05/05/11 13:43, Nick Jones wrote: This is on an Apple MacBook Air with SSD, and mutt has been configured with header-cacheing enabled and support for tokyo-cabinet compiled therein. For me this is still too slow, but I'm not sure that there's much else that can be done to improve it. According to tokyocabinet's homepage [1], they recomment kyotocabinet [2]. I don't know how easy it would be to incorporate in mutt or how much better (faster) it is. Quote: BTW, do you know Kyoto Cabinet? Actually, it is more powerful and convenient library than Tokyo Cabinet. At this distance of time, Kyoto Cabinet surpasses Tokyo Cabinet in every aspects. I strongly recommend you to use Kyoto Cabinet. [1] http://1978th.net/tokyocabinet/ [2] http://fallabs.com/kyotocabinet/ pgpIPwPK8NXSf.pgp Description: PGP signature
Re: offlineimap much slower than gmail-imap
On May 05, 2011 at 01:43 PM +0100, Nick Jones wrote: For reference, mutt (1.5.20) on my machine currently takes 11 seconds to open my offlineimap'd Gmail 'All Mail' folder which contains 17,418 messages. It then takes a further 6 seconds to close the mailbox, write any changes, and then switch back to my Inbox (for example). I've had different experiences with different folders on my Macbook (no SSD). Some large folders with 10-20k messages open up in seconds. Others can take a long time. I'm not sure what the issue is. I sent a message to the list about it a couple months ago but no one had any ideas. I don't know if the header/body caches are cleaned out properly or what, but the one time I nuked my cache, things really sped up for a couple weeks. This is all on local maildirs.
Re: offlineimap much slower than gmail-imap
My header cache (tokyo cabinet) seems to get slow on certain mailboxes every few weeks. I just blow away that mailboxes cache and let it rebuild and all is well again. I ALWAYS blow away the entire cache whenever I pull a new version of mutt from mercurial and whenever tokyo cabinet gets updated (Mac OSX). I don't know if this will help for offlineimap, or not. festus On Thu, 05 May 2011 10:35 -0400, Tim Gray lists+m...@protozoic.com wrote: On May 05, 2011 at 01:43 PM +0100, Nick Jones wrote: For reference, mutt (1.5.20) on my machine currently takes 11 seconds to open my offlineimap'd Gmail 'All Mail' folder which contains 17,418 messages. It then takes a further 6 seconds to close the mailbox, write any changes, and then switch back to my Inbox (for example). I've had different experiences with different folders on my Macbook (no SSD). Some large folders with 10-20k messages open up in seconds. Others can take a long time. I'm not sure what the issue is. I sent a message to the list about it a couple months ago but no one had any ideas. I don't know if the header/body caches are cleaned out properly or what, but the one time I nuked my cache, things really sped up for a couple weeks. This is all on local maildirs.
Re: offlineimap much slower than gmail-imap
On May 05, 2011 at 08:52 AM -0600, John J. Foster wrote: My header cache (tokyo cabinet) seems to get slow on certain mailboxes every few weeks. I just blow away that mailboxes cache and let it rebuild and all is well again. I ALWAYS blow away the entire cache whenever I pull a new version of mutt from mercurial and whenever tokyo cabinet gets updated (Mac OSX). Maybe I'll try doing that more often. I'm using tokyo cabinet as well. It's strange - my inbox which usually only has ~200 messages in it opens slower than some mailing list boxes with 1000's of messages. However, there's a lot more turnover in my inbox; it's rarely the same 200 messages.
Re: offlineimap much slower than gmail-imap
Forgot to mention - this is strictly an IMAP connection - no local mailboxes On Thu, 05 May 2011 11:12 -0400, Tim Gray lists+m...@protozoic.com wrote: On May 05, 2011 at 08:52 AM -0600, John J. Foster wrote: My header cache (tokyo cabinet) seems to get slow on certain mailboxes every few weeks. I just blow away that mailboxes cache and let it rebuild and all is well again. I ALWAYS blow away the entire cache whenever I pull a new version of mutt from mercurial and whenever tokyo cabinet gets updated (Mac OSX). Maybe I'll try doing that more often. I'm using tokyo cabinet as well. It's strange - my inbox which usually only has ~200 messages in it opens slower than some mailing list boxes with 1000's of messages. However, there's a lot more turnover in my inbox; it's rarely the same 200 messages.
Re: offlineimap much slower than gmail-imap
Hi again, Thanks for your help: I finally decided to use a database for my searches. Just installed mairix and made a simple script: everytime I need to search, just swich to terminal, run a script that waits for the string I am searching, executes mairix and opens a new mutt sesion within the new output folder in just some few seconds. This is OK for me, so I will forget for a while about the I/O performance of my system. Thanks, -- jm PS: will try new search databases, like mu next hollydays
Re: offlineimap much slower than gmail-imap
On Mon, May 02, 2011 at 12:03:46AM +0200, Jose M Vidal wrote: I was happily using mutt with gmail-imap. Then I decided to switch to offlineimap+msmtp, so I could still use mutt offline, have a backup of all my e-mails and, hopefully, increase mutt speed by working locally. But, after having all installed an all my e-mails already downloaded (10GB / 45.000 e-mails), every time I switch from INBOX to All Mail, it takes 6 minutes (!) to refresh the index. I had caché enabled with imap, and I am stilll keeping it in my .muttrc: set header_cache=~/.mutt/GMail/cache/headers¬ set message_cachedir=~/.mutt/GMail/cache/bodies¬ But I am afraid cache only works with online imap, because now accessing to folders is much-much slower than with previous online configuration. Is there anything I can do to speed-up my mutt? Thanks in advanced, -- jm PS: my mutt is 1.5.20 If your maildir is on an ecryptfs filesystem, the problem is the stat() call that mutt makes to check for changes in the maildir files. This is very slow on an ecryptfs filesystem because the header on every file must be decrypted for the stat() call. If you are certain that no other program might modify your mail files, you can unset maildir_header_cache_verify to bypass the stat() call. I use this for my large archive folders (which I also set read-only for safety) with a folder hook: folder-hook 'archive' 'push toggle-write; unset maildir_header_cache_verify' After I did this, access to my large folders became virtually instant. -- Scott Barker sc...@mostlylinux.ca Linux Consultant http://www.mostlylinux.ca/scott
Re: offlineimap much slower than gmail-imap
Hi everybody. Thank you very much for your help. The situation now is: 1- As my /home folder is ecrypt I added to my .muttrc your suggestion (folder-hook 'archive' 'push toggle-write; unset maildir_header_cache_verify') Apparently, after a first refresh, the update of files looks inmediate, but after using mutt a little bit, it turns slower againg. The end result is that there's not a significant improve in performance: Sometimes it is very fast, sometimes it is very slow. 2- I checked my data, ang got this: jm@jm-ThinkPad-X200s:~/.mail/GMail/[Gmail].All Mail$ du -h 4,0K./tmp 940K./new 9,4G./cur 9,4G. jm@jm-ThinkPad-X200s:~/.mail/GMail/[Gmail].All Mail$ find . -type f | wc -l 47860 The data shown in my mutt status page is the same (I am using google apps, premium, so nothing strange about having more than 7Gb) 3- My system is 32 bit. jm@jm-ThinkPad-X200s:~$ uname -m i686 4- After working a little bit with I/O, I get: TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND 120 4886 be/4 jm 2.38 M/s0.00 B/s 0.00 % 96.43 % mutt Honestly speaking, I don't know what to do with this information. Thanks again for your help. jm PS: will keep tokyo-cabinet for later, thanks. Think that understanding what is happenings is neccessary first.
Re: offlineimap much slower than gmail-imap
What does the following command give you (assuming your disk is /dev/sda): hdparm -tT /dev/sda jm@jm-ThinkPad-X200s:~$ sudo hdparm -tT /dev/sda6 /dev/sda6: Timing cached reads: 2738 MB in 2.00 seconds = 1370.52 MB/sec Timing buffered disk reads: 116 MB in 3.02 seconds = 38.45 MB/sec PS: note I had to sudo it. My /home doesn't have a specific partition this time: jm@jm-ThinkPad-X200s:~$ df -h S.archivosTam. Usado Disp. % Uso Montado en /dev/sda6 45G 29G 14G 69% / none 950M 320K 950M 1% /dev none 956M 1,7M 954M 1% /dev/shm none 956M 100K 955M 1% /var/run none 956M 0 956M 0% /var/lock /home/jm/.Private 45G 29G 14G 69% /home/jm Thanks again, -- jm
Re: offlineimap much slower than gmail-imap
On Mon, May 02, 2011 at 12:03:46AM +0200, Jose M Vidal wrote: I was happily using mutt with gmail-imap. Then I decided to switch to offlineimap+msmtp, so I could still use mutt offline, have a backup of all my e-mails and, hopefully, increase mutt speed by working locally. But, after having all installed an all my e-mails already downloaded (10GB / 45.000 e-mails), every time I switch from INBOX to All Mail, it takes 6 minutes (!) to refresh the index. I had caché enabled with imap, and I am stilll keeping it in my .muttrc: set header_cache=~/.mutt/GMail/cache/headers¬ set message_cachedir=~/.mutt/GMail/cache/bodies¬ But I am afraid cache only works with online imap, because now accessing to folders is much-much slower than with previous online configuration. Is there anything I can do to speed-up my mutt? Thanks in advanced, -- jm PS: my mutt is 1.5.20 I'm using offlineimap for severral accounts (one of them with ~5 mails) and refreshing is very fast. I only have: set header_cache='~/.mutt/hcache because it is not necessary to cache whole massage (they are already on your local disk). Sory I can't help any more, but just wanted to say that offlineimap works fine with mutt. Your problem must be elsewhere. Regards, -- Veljko
Re: offlineimap much slower than gmail-imap
No way: just set the caching just for headers, but the response is still very slow. Any other clue I can follow? Thanks a lot! jm
Re: offlineimap much slower than gmail-imap
Hi, On Mon, May 02, 2011 at 02:17:26PM +0200, Jose M Vidal wrote: No way: just set the caching just for headers, but the response is still very slow. Any other clue I can follow? Thanks a lot! as offlineimap is working fast for me as well I can only give you the generic hint to hunt down the problem with generic performance troubleshooting tools such as top, mpstat, iostat and posibly tcpdump and lsof as well. hth thomas
Re: offlineimap much slower than gmail-imap
On Mon, May 02, 2011 at 02:17:26PM +0200, Jose M Vidal wrote: No way: just set the caching just for headers, but the response is still very slow. Any other clue I can follow? Thanks a lot! jm My best guess is I/O load. If I use rtorrent with 10 torrents running (each of them with lots of connections that read/write to disk) it does takes lot more time to index 5 mails. iotop is nice tool for I/O troubleshooting. Maybe you should start there. Regards, -- Veljko
Re: offlineimap much slower than gmail-imap
On Mon, May 02, 2011 at 02:17:26PM +0200, Jose M Vidal wrote: No way: just set the caching just for headers, but the response is still very slow. Any other clue I can follow? I use offlineimap with mutt, and found that switching to a maildir containing of the order of 10,000 emails was a somewhat slow (taking perhaps a minute or so), even with maildir header caching configured. I recompiled mutt with tokyo-cabinet support (which allows mutt to use a tokyo cabinet database for the backend header-cache database), and opening the same maildir now takes a couple of seconds at most. If nothing else appears to be the cause of your speed issues, you might like to try that. HTH, Toby -- Dr T. S. Cubitt Mathematics and Quantum Information group Department of Mathematics Complutense University Madrid, Spain email: ts...@cantab.net web: www.dr-qubit.org
