Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
My mail server is self-hosted. I think I would rather integrate Apache solr with Dovecot than transferring mailboxes from the server onto my laptop. Thank you for your suggestion though. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 22:23, Sam Kuper wrote: > > On Wed, Apr 20, 2022 at 09:55:02PM +0800, Turritopsis Dohrnii Teo En Ming > wrote: > >> My advice for anyone who wants to be able to keep and search very large > >> monolithic mailstores would be: synchronise (e.g. via > >> https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison > >> or whatever) those mailstores from the server onto your local > >> filesystem; use maildir on your local filesystem; and use either Mutt's > >> "limiting" functions, or notmuch's index/search functions, for > >> searching/browsing. > > > > Wouldn't it be very tedious and time consuming to sync mailboxes from > > the server onto our local filesystems? > > Setting it up can be tedious depending on your needs. > > If your mailserver is self-hosted, giving you direct access to the > filesystem, that gives you more options (rsync, Unison, Dovecot dsync, > etc). > > Even if not, it's still possible using isync, fetchmail, or whatever > (see URL above). > > Once the initial sync is performed, though, subsequent syncs should only > need to transmit the difference between the local and remote mailstores, > and therefore should usually be fast and, optionally, automatic. > > Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Wed, Apr 20, 2022 at 09:55:02PM +0800, Turritopsis Dohrnii Teo En Ming wrote: >> My advice for anyone who wants to be able to keep and search very large >> monolithic mailstores would be: synchronise (e.g. via >> https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison >> or whatever) those mailstores from the server onto your local >> filesystem; use maildir on your local filesystem; and use either Mutt's >> "limiting" functions, or notmuch's index/search functions, for >> searching/browsing. > > Wouldn't it be very tedious and time consuming to sync mailboxes from > the server onto our local filesystems? Setting it up can be tedious depending on your needs. If your mailserver is self-hosted, giving you direct access to the filesystem, that gives you more options (rsync, Unison, Dovecot dsync, etc). Even if not, it's still possible using isync, fetchmail, or whatever (see URL above). Once the initial sync is performed, though, subsequent syncs should only need to transmit the difference between the local and remote mailstores, and therefore should usually be fast and, optionally, automatic. Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
> However, Gmail's web interface, and Gmail's proprietary mobile apps, > almost certainly do *not* use IMAP2 to communicate with the Gmail > servers. As has already been pointed out in this thread, they likely > use a proprietary protocol optimised for that proprietary environment. Noted and taken note of. Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 22:16, Sam Kuper wrote: > > On Wed, Apr 20, 2022 at 09:51:18PM +0800, Turritopsis Dohrnii Teo En Ming > wrote: > > I believe Gmail is using IMAP. The instructions for configuring Gmail > > email accounts in Outlook specifically mention IMAP server hostname: > > > > imap.gmail.com > > > > TCP Port 993, SSL > > Gmail offers an IMAP2 endpoint so that users can connect to Gmail with > IMAP2 clients (Thunderbird, Claws, Apple's Mail.app, or whatever) if > they wish to. > > However, Gmail's web interface, and Gmail's proprietary mobile apps, > almost certainly do *not* use IMAP2 to communicate with the Gmail > servers. As has already been pointed out in this thread, they likely > use a proprietary protocol optimised for that proprietary environment. > > Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
I would prefer to use Outlook as I have Microsoft Office desktop apps. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 22:12, Shawn Heisey wrote: > > On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote: > > I believe Gmail is using IMAP. The instructions for configuring Gmail > > email accounts in Outlook specifically mention IMAP server hostname: > > > > imap.gmail.com > > > > TCP Port 993, SSL > > Indeed you CAN use IMAP with gmail. For work, I mostly use my email > (which is gmail-based) via Thunderbird, with IMAP. I am familiar with it. > > But if I use the Gmail app for Android or the gmail website, I feel > absolutely certain that those applications are NOT using IMAP. Chances > are that they use protocols developed by Google. > > Regarding your next email on this thread, the initial sync of a large > number of messages via IMAP does take a really long time. But once the > initial sync is complete, syncing the differences is very fast. The big > problem with accessing a large folder via a webmail app like Roundcube > is that it never sets up a local copy of the mailbox. I actually don't > know where the copy of all those message headers resides ... whether it > is in the PHP interpreter or in the browser. But it's not persisted to > disk as it would be with a more traditional IMAP client such as Thunderbird. > > Thanks, > Shawn >
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Wed, Apr 20, 2022 at 09:51:18PM +0800, Turritopsis Dohrnii Teo En Ming wrote: > I believe Gmail is using IMAP. The instructions for configuring Gmail > email accounts in Outlook specifically mention IMAP server hostname: > > imap.gmail.com > > TCP Port 993, SSL Gmail offers an IMAP2 endpoint so that users can connect to Gmail with IMAP2 clients (Thunderbird, Claws, Apple's Mail.app, or whatever) if they wish to. However, Gmail's web interface, and Gmail's proprietary mobile apps, almost certainly do *not* use IMAP2 to communicate with the Gmail servers. As has already been pointed out in this thread, they likely use a proprietary protocol optimised for that proprietary environment. Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote: I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname: imap.gmail.com TCP Port 993, SSL Indeed you CAN use IMAP with gmail. For work, I mostly use my email (which is gmail-based) via Thunderbird, with IMAP. I am familiar with it. But if I use the Gmail app for Android or the gmail website, I feel absolutely certain that those applications are NOT using IMAP. Chances are that they use protocols developed by Google. Regarding your next email on this thread, the initial sync of a large number of messages via IMAP does take a really long time. But once the initial sync is complete, syncing the differences is very fast. The big problem with accessing a large folder via a webmail app like Roundcube is that it never sets up a local copy of the mailbox. I actually don't know where the copy of all those message headers resides ... whether it is in the PHP interpreter or in the browser. But it's not persisted to disk as it would be with a more traditional IMAP client such as Thunderbird. Thanks, Shawn
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
> My advice for anyone who wants to be able to keep and search very large > monolithic mailstores would be: synchronise (e.g. via > https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison > or whatever) those mailstores from the server onto your local > filesystem; use maildir on your local filesystem; and use either Mutt's > "limiting" functions, or notmuch's index/search functions, for > searching/browsing. Wouldn't it be very tedious and time consuming to sync mailboxes from the server onto our local filesystems? Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 14:21, Sam Kuper wrote: > > On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote: > > I would bet that if you accessed a gmail folder with 5 million > > messages in it using IMAP, you would have similar problems with it to > > those that have been described here in this thread. IMAP is a > > beautiful protocol, but I don't think it was designed for handling > > that many messages. > > This. Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with > us to confirm. > > Even at just ~100B for each message's headers, your IMAP2 client would > likely need at least ~500MB free RAM to load 5 million mails. > > By the time Mark stopped working on UW IMAP (the reference IMAP > implementation, aka Panda IMAP), c.2010, even top-of-the-range > smartphones typically had only ~512MB RAM total, and top-of-the-range > ThinkPads had max ~4GiB (which was the upper limit of what 32-bit > operating systems, still popular then, could handle). > > When IMAP2 was invented, c.1988-1990, RAM like that was basically > supercomputer territory. > > Had Mark intended or expected IMAP2 users to have had supercomputers at > their disposal, and to be searching such large volumes of mail over the > protocol, I suspect he would have designed the protocol differently: for > raw efficiency over human readability. > > My advice for anyone who wants to be able to keep and search very large > monolithic mailstores would be: synchronise (e.g. via > https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison > or whatever) those mailstores from the server onto your local > filesystem; use maildir on your local filesystem; and use either Mutt's > "limiting" functions, or notmuch's index/search functions, for > searching/browsing. > > Good luck in your quest! > > Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname: imap.gmail.com TCP Port 993, SSL Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 09:26, Shawn Heisey wrote: > > On 4/19/2022 5:12 PM, Turritopsis Dohrnii Teo En Ming wrote: > > It is not true that no email system was ever designed for 200,000 or > > more emails. > > > > If I did not remember wrongly, more than 10 years ago, I used to have > > a Gmail email account, with 4-5 MILLION email messages in the Inbox. > > There was no problem searching even though I had 4-5 MILLION email > > messages in the Inbox. > > If you were using either a Google app or the gmail.com website to access > that inbox, then it doesn't compare. I really doubt that either of > those uses the IMAP protocol. They would be using something proprietary > that is highly optimized for the way that Google stores data and > leverages the enormous amounts of computing power that they maintain. > > I would bet that if you accessed a gmail folder with 5 million messages > in it using IMAP, you would have similar problems with it to those that > have been described here in this thread. IMAP is a beautiful protocol, > but I don't think it was designed for handling that many messages. > > Thanks, > Shawn >
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote: > I would bet that if you accessed a gmail folder with 5 million > messages in it using IMAP, you would have similar problems with it to > those that have been described here in this thread. IMAP is a > beautiful protocol, but I don't think it was designed for handling > that many messages. This. Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with us to confirm. Even at just ~100B for each message's headers, your IMAP2 client would likely need at least ~500MB free RAM to load 5 million mails. By the time Mark stopped working on UW IMAP (the reference IMAP implementation, aka Panda IMAP), c.2010, even top-of-the-range smartphones typically had only ~512MB RAM total, and top-of-the-range ThinkPads had max ~4GiB (which was the upper limit of what 32-bit operating systems, still popular then, could handle). When IMAP2 was invented, c.1988-1990, RAM like that was basically supercomputer territory. Had Mark intended or expected IMAP2 users to have had supercomputers at their disposal, and to be searching such large volumes of mail over the protocol, I suspect he would have designed the protocol differently: for raw efficiency over human readability. My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing. Good luck in your quest! Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On 4/19/2022 5:12 PM, Turritopsis Dohrnii Teo En Ming wrote: It is not true that no email system was ever designed for 200,000 or more emails. If I did not remember wrongly, more than 10 years ago, I used to have a Gmail email account, with 4-5 MILLION email messages in the Inbox. There was no problem searching even though I had 4-5 MILLION email messages in the Inbox. If you were using either a Google app or the gmail.com website to access that inbox, then it doesn't compare. I really doubt that either of those uses the IMAP protocol. They would be using something proprietary that is highly optimized for the way that Google stores data and leverages the enormous amounts of computing power that they maintain. I would bet that if you accessed a gmail folder with 5 million messages in it using IMAP, you would have similar problems with it to those that have been described here in this thread. IMAP is a beautiful protocol, but I don't think it was designed for handling that many messages. Thanks, Shawn
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
The emails appeared in my Gmail inbox. Then sometime later the emails disappeared mysteriously. Not sure who deleted my emails from my Gmail email account. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 06:42, Joseph Tam wrote: > > On Tue, 19 Apr 2022, Sami Ketola wrote: > > >>> In practice, though, Gmail used to exhibit search/browse bugs (e.g. > >>> failing to identify all relevant mails/threads) annoyingly often. This > >>> was sometimes true via the official Gmail web interface, and was > >>> especially true via the official mobile app, on at least some platforms. > >> > >> They are also losing/deleting messages. I have had several occassions > >> where I send a message and my delivering server registered the Sent > >> status of the gmail server and it just disappeared. > > > > This is more about gmail ?spam protection?. Gmail filters emails after > > accepting them with secret criteria and sometimes the email ends up in > > spam folder and sometime it is just silently deleted. > > While Gmail's spam classifcation is frustratingly opaque (as with many > other providers), there's one other weirdness that can make messages > dissappear: when you send from Gmail through a forwarder back to the > same Gmail account. > > The mail will be accepted, but not appear in your INBOX. This is either > some labelling weirdness (cannot be labelled as both INBOX and Sent) or > maybe an anti-loop protection. This causes confusion when Gmail users > test their mail forwarding I set up for them. > > However, we're truly off-topic: my point was that Gmail users get a > distorted sense of how most mail systems work (mailbox operations scale > with message count), as they get brainwashed into the "Gmail" way of > doing things. A relevant question is whether you can use Dovecot's > virtual mailbox feature to define a catch-all virtual mailbox to placate > these users which won't bring an imap process to its knees. > > Joseph Tam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
For me, it is not because of Gmail spam filters. I confirm I had the emails in my inbox previously, and then sometime later the emails were deleted mysteriously. Not sure who deleted my emails. Definitely not because of Gmail spam filters. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wed On Tue, 19 Apr 2022 at 16:03, Sami Ketola wrote: > > > > > On 19. Apr 2022, at 10.36, Marc wrote: > > > >> Gmail is *advertised* as working that way. > >> > >> In practice, though, Gmail used to exhibit search/browse bugs (e.g. > >> failing to identify all relevant mails/threads) annoyingly often. This > >> was sometimes true via the official Gmail web interface, and was > >> especially true via the official mobile app, on at least some platforms. > >> > > > > They are also losing/deleting messages. I have had several occassions where > > I send a message and my delivering server registered the Sent status of the > > gmail server and it just disappeared. > > This is more about gmail “spam protection”. Gmail filters emails after > accepting them with secret criteria and sometimes the email ends up in spam > folder and sometime it is just silently deleted. > > By using gmail there is no way of telling how many (important) emails you > have not received because of this. > > Sami >
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
Recently, I discovered that some emails in one of my Gmail email accounts were missing. Not sure if Gmail/Google had deleted my email messages. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Tue, 19 Apr 2022 at 15:36, Marc wrote: > > > Gmail is *advertised* as working that way. > > > > In practice, though, Gmail used to exhibit search/browse bugs (e.g. > > failing to identify all relevant mails/threads) annoyingly often. This > > was sometimes true via the official Gmail web interface, and was > > especially true via the official mobile app, on at least some platforms. > > > > They are also losing/deleting messages. I have had several occassions where I > send a message and my delivering server registered the Sent status of the > gmail server and it just disappeared. > > > > > In any case, for Dovecot, Joseph & Paul's advice to divide mail into > > folders where possible seems sensible - especially for access over IMAP. > > > > I also made some cronjobs, sieve scripts that archive/move message the most > used folders/mailboxes to archive/2021, archive/2020 etc. >
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
I am planning to delete a lot of useless emails, which takes up a huge proportion of the 200,000 emails in my inbox. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Tue, 19 Apr 2022 at 14:49, Sam Kuper wrote: > > On Mon, Apr 18, 2022 at 03:23:10PM -0700, Joseph Tam wrote: > > On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote: > >> As for the 200,000+ emails in the inbox no email system was ever > >> designed for that - ever. > >> ... > >> no system will support 200,000 + emails, even if the server can > >> handle that and running imap where you only download the headers the > >> email client would just spin trying to update the email box > >> constantantly. > > > > I think Gmail does exaclty this -- their mail system really has one > > big mesage repository, and they simulate mailboxes by using labels. > > They seem to encourage piling the message high and using their search > > or auto-labelling features to find what you're looking for. > > > > Users of mine who previously used Gmail expect our mail system to > > behave similarly, and I have to break them of their habit to packrat > > all their mail into their INBOX. > > Gmail is *advertised* as working that way. > > In practice, though, Gmail used to exhibit search/browse bugs (e.g. > failing to identify all relevant mails/threads) annoyingly often. This > was sometimes true via the official Gmail web interface, and was > especially true via the official mobile app, on at least some platforms. > > Maybe those issues have been fixed - I don't know. But unless they have > been, Gmail is not really a panacea. > > In any case, for Dovecot, Joseph & Paul's advice to divide mail into > folders where possible seems sensible - especially for access over IMAP. > > Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
I used to have 4-5 MILLION email messages in my Gmail email account, more than 10 years ago. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Tue, 19 Apr 2022 at 06:23, Joseph Tam wrote: > > On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote: > > > As for the 200,000+ emails in the inbox no email system was ever designed > > for > > that - ever. > >... > > no system will support 200,000 + emails, even if the server can handle that > > and running imap where you only download the headers the email client would > > just spin trying to update the email box constantantly. > > I think Gmail does exaclty this -- their mail system really has one > big mesage repository, and they simulate mailboxes by using labels. > They seem to encourage piling the message high and using their search > or auto-labelling features to find what you're looking for. > > Users of mine who previously used Gmail expect our mail system to behave > similarly, and I have to break them of their habit to packrat all their > mail into their INBOX. > > Joseph Tam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
It is not true that no email system was ever designed for 200,000 or more emails. If I did not remember wrongly, more than 10 years ago, I used to have a Gmail email account, with 4-5 MILLION email messages in the Inbox. There was no problem searching even though I had 4-5 MILLION email messages in the Inbox. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Mon, 18 Apr 2022 at 17:44, Paul Kudla (SCOM.CA Internet) wrote: > > this is not the answer you are looking for but comes from years and years of > experience with mail servers in general (cyrus first then dovecot) > > First off imap clients can handle emails counts but vary on performance > > 1. outlook for example is completely useless above 1000 count per folder/sub > folder, imap was kinda supported to outlook 2010 but after 2012 fell off > largely due to microsoft pushing their sync & exchange servers. > > 2. thunderbird will and can handle 1~3 emails but anything above > 1 really should be split into a subfolder, i usually take the inbox and > split into the year of the messages to keep into a subfolder like 2020,2021 > etc > > 3. apple mail and iphone related seem to handle large emails boxes ok but > will slow down above the 1000+ per folder limit > > 4. android you need to run bluemail or something like that but again limits > are like the iphone for around 1000 per folder, generic imap client does not > work well with android. > > As for the 200,000+ emails in the inbox no email system was ever designed for > that - ever. > > It needs to be understood that pending how your server hdd's are formatted > (nfs, ufs, zfs etc) will directly effect the speed of email retreivals. > > nfs being the slowest (dependant on network speeds, server speeds etc) > > ufs is the old style unix partitions, ufs works ok but you need to tweek > certain settings for how many inodes can be in memory at any one time to i > believe a max of 10,000 (freebsd, other systems will vary) > > zfs is the best as it will store dir inode etc into memory in the system so > large file numbers in directories have a faster chance to respond, again > system dependant suggest lots of memory > > no system will support 200,000 + emails, even if the server can handle that > and running imap where you only download the headers the email client would > just spin trying to update the email box constantantly. > > forget about trying to index searches etc. > > internet speed (even a 100M fibre connection) is also a factor. > > this is a sample dir structure from my server > > 1 email = 1 file in a sub dir. > > the operating system is where you are choking on its ability to handle > 200,000 files in one directory is just unsustainable. > > if you have to handle that many incoming emails i strongly suggest a sieve > filter that automatically moves incoming emails into a folder based on the > email to keep the counts under control. > > I find dovecot can handle email counts better then the old cyrus but when i > ran cyrus i was using ufs file system as well, when i upgraded to dovecot i > had also upgraded to zfs. > > # cd cur > > [05:39:14] mail18.scom.ca [root:0] > /data/dovecot/users/scom.ca/p...@scom.ca/cur > # ll > total 77333 > drwx--2 vmail vmail uarch 361B Apr 18 05:35 . > drwx-- 947 vmail vmail uarch 963B Apr 18 05:36 .. > -rw---1 vmail vmail uarch 6.4K Feb 20 08:05 > 1645362302.M163606P37919.mail18.scom.ca,S=6536,W=6687:2,FS > -rw---1 vmail vmail uarch 8.9K Feb 20 08:05 > 1645362302.M535115P37919.mail18.scom.ca,S=9074,W=9224:2,S > -rw---1 vmail vmail uarch 5.5K Feb 20 08:05 > 1645362303.M133625P37919.mail18.scom.ca,S=5632,W=5729:2,S > -rw---1 vmail vmail uarch 11K Feb 20 08:05 > 1645362305.M954336P38787.mail18.scom.ca,S=11438,W=11642:2,S > -rw---1 vmail vmail uarch 1.9K Feb 20 08:05 > 1645362305.M954355P38787.mail18.scom.ca,S=1912,W=1965:2,S > -rw---1 vmail vmail uarch 2.3K Feb 20 08:05 > 1645362306.M417932P37919.mail18.scom.ca,S=2333,W=2413:2,S > -rw---1 vmail vmail uarch 2.2K Feb 20 08:05 > 1645362306.M569866P37919.mail18.scom.ca,S=2287,W=2366:2,S > -rw---1 vmail vmail uarch 2.3K Feb 20 08:05 > 1645362307.M184339P37919.mail18.scom.ca,S=2371,W=2452:2,S > -rw---1 vmail vmail uarch 22K Feb 20 08:05 > 1645362307.M706237P37919.mail18.scom.ca,S=22281,W=22692:2,S > -rw---1 vmail vmail uarch 189K Feb 20 08:05 > 1645362313.M247319P37919.mail18.scom.ca,S=193354,W=196073:2,S > -rw---1 vmail vmail uarch 182K Feb 20 08:05 > 1645362313.M68190P39861.mail18.scom.ca,S=186263,W=10:2,S > > fyi > > > > > > > > > > Happy Monday !!! > Thanks - paul > > Paul Kudla > > > > 004-1009 Byron Street South > Whitby, Ontario - Canada > L1N 4S3 > > Toronto 416.642.7266 > Main 1.866.411.7266 > Fax 1.888.892.7266 > On 4/17/2022 5:19 AM, dovecot-boun...@dovecot.org
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Tue, 19 Apr 2022, Sami Ketola wrote: In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms. They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared. This is more about gmail ?spam protection?. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted. While Gmail's spam classifcation is frustratingly opaque (as with many other providers), there's one other weirdness that can make messages dissappear: when you send from Gmail through a forwarder back to the same Gmail account. The mail will be accepted, but not appear in your INBOX. This is either some labelling weirdness (cannot be labelled as both INBOX and Sent) or maybe an anti-loop protection. This causes confusion when Gmail users test their mail forwarding I set up for them. However, we're truly off-topic: my point was that Gmail users get a distorted sense of how most mail systems work (mailbox operations scale with message count), as they get brainwashed into the "Gmail" way of doing things. A relevant question is whether you can use Dovecot's virtual mailbox feature to define a catch-all virtual mailbox to placate these users which won't bring an imap process to its knees. Joseph Tam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
> On 19. Apr 2022, at 10.36, Marc wrote: > >> Gmail is *advertised* as working that way. >> >> In practice, though, Gmail used to exhibit search/browse bugs (e.g. >> failing to identify all relevant mails/threads) annoyingly often. This >> was sometimes true via the official Gmail web interface, and was >> especially true via the official mobile app, on at least some platforms. >> > > They are also losing/deleting messages. I have had several occassions where I > send a message and my delivering server registered the Sent status of the > gmail server and it just disappeared. This is more about gmail “spam protection”. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted. By using gmail there is no way of telling how many (important) emails you have not received because of this. Sami
RE: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
> Gmail is *advertised* as working that way. > > In practice, though, Gmail used to exhibit search/browse bugs (e.g. > failing to identify all relevant mails/threads) annoyingly often. This > was sometimes true via the official Gmail web interface, and was > especially true via the official mobile app, on at least some platforms. > They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared. > > In any case, for Dovecot, Joseph & Paul's advice to divide mail into > folders where possible seems sensible - especially for access over IMAP. > I also made some cronjobs, sieve scripts that archive/move message the most used folders/mailboxes to archive/2021, archive/2020 etc.
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Mon, Apr 18, 2022 at 03:23:10PM -0700, Joseph Tam wrote: > On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote: >> As for the 200,000+ emails in the inbox no email system was ever >> designed for that - ever. >> ... >> no system will support 200,000 + emails, even if the server can >> handle that and running imap where you only download the headers the >> email client would just spin trying to update the email box >> constantantly. > > I think Gmail does exaclty this -- their mail system really has one > big mesage repository, and they simulate mailboxes by using labels. > They seem to encourage piling the message high and using their search > or auto-labelling features to find what you're looking for. > > Users of mine who previously used Gmail expect our mail system to > behave similarly, and I have to break them of their habit to packrat > all their mail into their INBOX. Gmail is *advertised* as working that way. In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms. Maybe those issues have been fixed - I don't know. But unless they have been, Gmail is not really a panacea. In any case, for Dovecot, Joseph & Paul's advice to divide mail into folders where possible seems sensible - especially for access over IMAP. Sam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote: As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. ... no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly. I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for. Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX. Joseph Tam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
this is not the answer you are looking for but comes from years and years of experience with mail servers in general (cyrus first then dovecot) First off imap clients can handle emails counts but vary on performance 1. outlook for example is completely useless above 1000 count per folder/sub folder, imap was kinda supported to outlook 2010 but after 2012 fell off largely due to microsoft pushing their sync & exchange servers. 2. thunderbird will and can handle 1~3 emails but anything above 1 really should be split into a subfolder, i usually take the inbox and split into the year of the messages to keep into a subfolder like 2020,2021 etc 3. apple mail and iphone related seem to handle large emails boxes ok but will slow down above the 1000+ per folder limit 4. android you need to run bluemail or something like that but again limits are like the iphone for around 1000 per folder, generic imap client does not work well with android. As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. It needs to be understood that pending how your server hdd's are formatted (nfs, ufs, zfs etc) will directly effect the speed of email retreivals. nfs being the slowest (dependant on network speeds, server speeds etc) ufs is the old style unix partitions, ufs works ok but you need to tweek certain settings for how many inodes can be in memory at any one time to i believe a max of 10,000 (freebsd, other systems will vary) zfs is the best as it will store dir inode etc into memory in the system so large file numbers in directories have a faster chance to respond, again system dependant suggest lots of memory no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly. forget about trying to index searches etc. internet speed (even a 100M fibre connection) is also a factor. this is a sample dir structure from my server 1 email = 1 file in a sub dir. the operating system is where you are choking on its ability to handle 200,000 files in one directory is just unsustainable. if you have to handle that many incoming emails i strongly suggest a sieve filter that automatically moves incoming emails into a folder based on the email to keep the counts under control. I find dovecot can handle email counts better then the old cyrus but when i ran cyrus i was using ufs file system as well, when i upgraded to dovecot i had also upgraded to zfs. # cd cur [05:39:14] mail18.scom.ca [root:0] /data/dovecot/users/scom.ca/p...@scom.ca/cur # ll total 77333 drwx-- 2 vmail vmail uarch 361B Apr 18 05:35 . drwx-- 947 vmail vmail uarch 963B Apr 18 05:36 .. -rw--- 1 vmail vmail uarch 6.4K Feb 20 08:05 1645362302.M163606P37919.mail18.scom.ca,S=6536,W=6687:2,FS -rw--- 1 vmail vmail uarch 8.9K Feb 20 08:05 1645362302.M535115P37919.mail18.scom.ca,S=9074,W=9224:2,S -rw--- 1 vmail vmail uarch 5.5K Feb 20 08:05 1645362303.M133625P37919.mail18.scom.ca,S=5632,W=5729:2,S -rw--- 1 vmail vmail uarch 11K Feb 20 08:05 1645362305.M954336P38787.mail18.scom.ca,S=11438,W=11642:2,S -rw--- 1 vmail vmail uarch 1.9K Feb 20 08:05 1645362305.M954355P38787.mail18.scom.ca,S=1912,W=1965:2,S -rw--- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362306.M417932P37919.mail18.scom.ca,S=2333,W=2413:2,S -rw--- 1 vmail vmail uarch 2.2K Feb 20 08:05 1645362306.M569866P37919.mail18.scom.ca,S=2287,W=2366:2,S -rw--- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362307.M184339P37919.mail18.scom.ca,S=2371,W=2452:2,S -rw--- 1 vmail vmail uarch 22K Feb 20 08:05 1645362307.M706237P37919.mail18.scom.ca,S=22281,W=22692:2,S -rw--- 1 vmail vmail uarch 189K Feb 20 08:05 1645362313.M247319P37919.mail18.scom.ca,S=193354,W=196073:2,S -rw--- 1 vmail vmail uarch 182K Feb 20 08:05 1645362313.M68190P39861.mail18.scom.ca,S=186263,W=10:2,S fyi Happy Monday !!! Thanks - paul Paul Kudla 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 On 4/17/2022 5:19 AM, dovecot-boun...@dovecot.org wrote: On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote: Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox Good day from Singapore, I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany. [1st issue] === Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox. Roundcube webmail doesn't show any error. Search results simply don't show up after a long time. However, I got the following error in Squirrelmail webmail (using
