Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

My mail server is self-hosted. I think I would rather integrate Apache
solr with Dovecot than transferring mailboxes from the server onto my
laptop.

Thank you for your suggestion though.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday


On Wed, 20 Apr 2022 at 22:23, Sam Kuper  wrote:
>
> On Wed, Apr 20, 2022 at 09:55:02PM +0800, Turritopsis Dohrnii Teo En Ming 
> wrote:
> >> My advice for anyone who wants to be able to keep and search very large
> >> monolithic mailstores would be: synchronise (e.g. via
> >> https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison
> >> or whatever) those mailstores from the server onto your local
> >> filesystem; use maildir on your local filesystem; and use either Mutt's
> >> "limiting" functions, or notmuch's index/search functions, for
> >> searching/browsing.
> >
> > Wouldn't it be very tedious and time consuming to sync mailboxes from
> > the server onto our local filesystems?
>
> Setting it up can be tedious depending on your needs.
>
> If your mailserver is self-hosted, giving you direct access to the
> filesystem, that gives you more options (rsync, Unison, Dovecot dsync,
> etc).
>
> Even if not, it's still possible using isync, fetchmail, or whatever
> (see URL above).
>
> Once the initial sync is performed, though, subsequent syncs should only
> need to transmit the difference between the local and remote mailstores,
> and therefore should usually be fast and, optionally, automatic.
>
> Sam

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Sam Kuper]

On Wed, Apr 20, 2022 at 09:55:02PM +0800, Turritopsis Dohrnii Teo En Ming wrote:
>> My advice for anyone who wants to be able to keep and search very large
>> monolithic mailstores would be: synchronise (e.g. via
>> https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison
>> or whatever) those mailstores from the server onto your local
>> filesystem; use maildir on your local filesystem; and use either Mutt's
>> "limiting" functions, or notmuch's index/search functions, for
>> searching/browsing.
> 
> Wouldn't it be very tedious and time consuming to sync mailboxes from
> the server onto our local filesystems?

Setting it up can be tedious depending on your needs.

If your mailserver is self-hosted, giving you direct access to the
filesystem, that gives you more options (rsync, Unison, Dovecot dsync,
etc).

Even if not, it's still possible using isync, fetchmail, or whatever
(see URL above).

Once the initial sync is performed, though, subsequent syncs should only
need to transmit the difference between the local and remote mailstores,
and therefore should usually be fast and, optionally, automatic.

Sam

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

> However, Gmail's web interface, and Gmail's proprietary mobile apps,
> almost certainly do *not* use IMAP2 to communicate with the Gmail
> servers.  As has already been pointed out in this thread, they likely
> use a proprietary protocol optimised for that proprietary environment.

Noted and taken note of.

Thank you.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday


On Wed, 20 Apr 2022 at 22:16, Sam Kuper  wrote:
>
> On Wed, Apr 20, 2022 at 09:51:18PM +0800, Turritopsis Dohrnii Teo En Ming 
> wrote:
> > I believe Gmail is using IMAP. The instructions for configuring Gmail
> > email accounts in Outlook specifically mention IMAP server hostname:
> >
> > imap.gmail.com
> >
> > TCP Port 993, SSL
>
> Gmail offers an IMAP2 endpoint so that users can connect to Gmail with
> IMAP2 clients (Thunderbird, Claws, Apple's Mail.app, or whatever) if
> they wish to.
>
> However, Gmail's web interface, and Gmail's proprietary mobile apps,
> almost certainly do *not* use IMAP2 to communicate with the Gmail
> servers.  As has already been pointed out in this thread, they likely
> use a proprietary protocol optimised for that proprietary environment.
>
> Sam

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

I would prefer to use Outlook as I have Microsoft Office desktop apps.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday



On Wed, 20 Apr 2022 at 22:12, Shawn Heisey  wrote:
>
> On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote:
> > I believe Gmail is using IMAP. The instructions for configuring Gmail
> > email accounts in Outlook specifically mention IMAP server hostname:
> >
> > imap.gmail.com
> >
> > TCP Port 993, SSL
>
> Indeed you CAN use IMAP with gmail.  For work, I mostly use my email
> (which is gmail-based) via Thunderbird, with IMAP.  I am familiar with it.
>
> But if I use the Gmail app for Android or the gmail website, I feel
> absolutely certain that those applications are NOT using IMAP.  Chances
> are that they use protocols developed by Google.
>
> Regarding your next email on this thread, the initial sync of a large
> number of messages via IMAP does take a really long time. But once the
> initial sync is complete, syncing the differences is very fast.  The big
> problem with accessing a large folder via a webmail app like Roundcube
> is that it never sets up a local copy of the mailbox.  I actually don't
> know where the copy of all those message headers resides ... whether it
> is in the PHP interpreter or in the browser.  But it's not persisted to
> disk as it would be with a more traditional IMAP client such as Thunderbird.
>
> Thanks,
> Shawn
>

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Sam Kuper]

On Wed, Apr 20, 2022 at 09:51:18PM +0800, Turritopsis Dohrnii Teo En Ming wrote:
> I believe Gmail is using IMAP. The instructions for configuring Gmail
> email accounts in Outlook specifically mention IMAP server hostname:
> 
> imap.gmail.com
> 
> TCP Port 993, SSL

Gmail offers an IMAP2 endpoint so that users can connect to Gmail with
IMAP2 clients (Thunderbird, Claws, Apple's Mail.app, or whatever) if
they wish to.

However, Gmail's web interface, and Gmail's proprietary mobile apps,
almost certainly do *not* use IMAP2 to communicate with the Gmail
servers.  As has already been pointed out in this thread, they likely
use a proprietary protocol optimised for that proprietary environment.

Sam

Re: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

Dear Shawn Heisey,

Noted with thanks. I shall probably try it one of these coming weekends.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday


On Wed, 20 Apr 2022 at 22:10, Shawn Heisey  wrote:
>
> On 4/20/22 08:03, Turritopsis Dohrnii Teo En Ming wrote:
> > Dear Shawn Heisey,
> >
> > Your instructions for integrating Apache SOLR with Dovecot is too much
> > for me to understand. Do you have links/URLs to excellent guides for
> > doing this?
>
> Here's the dovecot manual page on it:
>
> https://doc.dovecot.org/configuration_manual/fts/solr/
>
> This doesn't explain how to tell dovecot to index new incoming messages,
> or how to initiate a full reindex, but I can guide you on those.
>
> Thanks,
> Shawn
>

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote:

I believe Gmail is using IMAP. The instructions for configuring Gmail
email accounts in Outlook specifically mention IMAP server hostname:

imap.gmail.com

TCP Port 993, SSL


Indeed you CAN use IMAP with gmail.  For work, I mostly use my email 
(which is gmail-based) via Thunderbird, with IMAP.  I am familiar with it.


But if I use the Gmail app for Android or the gmail website, I feel 
absolutely certain that those applications are NOT using IMAP.  Chances 
are that they use protocols developed by Google.


Regarding your next email on this thread, the initial sync of a large 
number of messages via IMAP does take a really long time. But once the 
initial sync is complete, syncing the differences is very fast.  The big 
problem with accessing a large folder via a webmail app like Roundcube 
is that it never sets up a local copy of the mailbox.  I actually don't 
know where the copy of all those message headers resides ... whether it 
is in the PHP interpreter or in the browser.  But it's not persisted to 
disk as it would be with a more traditional IMAP client such as Thunderbird.


Thanks,
Shawn

Re: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Shawn Heisey]

On 4/20/22 08:03, Turritopsis Dohrnii Teo En Ming wrote:

Dear Shawn Heisey,

Your instructions for integrating Apache SOLR with Dovecot is too much
for me to understand. Do you have links/URLs to excellent guides for
doing this?


Here's the dovecot manual page on it:

https://doc.dovecot.org/configuration_manual/fts/solr/

This doesn't explain how to tell dovecot to index new incoming messages, 
or how to initiate a full reindex, but I can guide you on those.


Thanks,
Shawn

Re: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

Dear Shawn Heisey,

Your instructions for integrating Apache SOLR with Dovecot is too much
for me to understand. Do you have links/URLs to excellent guides for
doing this?

Thank you.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday



On Mon, 18 Apr 2022 at 07:27, Shawn Heisey  wrote:
>
> On 4/17/2022 7:44 AM, Turritopsis Dohrnii Teo En Ming wrote:
> > My Dovecot, Postfix, Roundcube and Squirrelmail were automatically
> > installed, configured and deployed by Virtualmin web hosting control
> > panel install scripts.
> >
> > How do I integrate Apache SOLR searching capability with Dovecot? Are
> > there any excellent guides for this?
>
> I have this in /etc/dovecot/conf.d/10-mail.conf:
>
> mail_plugins = $mail_plugins quota fts fts_solr
>
> And in conf.d/90-plugin.conf, I have this:
>
> plugin {
>fts = solr
>fts_solr = url=http://localhost:8983/solr/dovecot/
>fts_autoindex = yes
>fts_enforced = yes
> }
>
> I'm running a version of Solr that I built myself from the branch_8_11
> source code -- 8.11.2-SNAPSHOT.  I installed that with the service
> installer script that it includes.  I have been customizing the index
> config, it started with the config and schema provided by dovecot, which
> did work.  I'm going to have some questions for those who write the
> fts_solr plugin, and I still need to glance at the source code.
>
> I am a committer on the Apache Solr project, so I do know a little bit
> about Solr. :)  I am not as competent in the C code that makes dovecot
> as I am in the Java code that makes Solr.  C was one of my first
> languages, but I have not spent any real time with it in decades.
>
> My install of dovecot has a grand total of 180079 messages in it right
> now, 154K of which are in my mailbox.  (I deleted the test folder I
> created to test your problem out).  If you can upgrade roundcube and
> dovecot, plus incorporate Solr, things might work better for you.
>
> I would suggest that you move most of the messages out of your inbox
> into some kind of archival folder.  Keep between 3 months and one year
> of messages in the actual inbox so working with that folder is fast, and
> you can visit the archive if you need to look at something older.
>
> I have never liked dealing with do-everything packages like webmin or
> virtualmin.  I want to understand each of the pieces that make up the
> whole, and be able to upgrade and reconfigure each piece independently
> from the others.  I find the configuration restraints of such a system
> very limiting.  Very often if you customize something yourself outside
> the GUI, you lose the ability to administer it in the GUI, because it
> doesn't know how to deal with the change.  Or if you do change something
> in the GUI afterwards, your customization might get deleted.
>
> Thanks,
> Shawn
>

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

> My advice for anyone who wants to be able to keep and search very large
> monolithic mailstores would be: synchronise (e.g. via
> https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison
> or whatever) those mailstores from the server onto your local
> filesystem; use maildir on your local filesystem; and use either Mutt's
> "limiting" functions, or notmuch's index/search functions, for
> searching/browsing.

Wouldn't it be very tedious and time consuming to sync mailboxes from
the server onto our local filesystems?

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday



On Wed, 20 Apr 2022 at 14:21, Sam Kuper  wrote:
>
> On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote:
> > I would bet that if you accessed a gmail folder with 5 million
> > messages in it using IMAP, you would have similar problems with it to
> > those that have been described here in this thread.  IMAP is a
> > beautiful protocol, but I don't think it was designed for handling
> > that many messages.
>
> This.  Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with
> us to confirm.
>
> Even at just ~100B for each message's headers, your IMAP2 client would
> likely need at least ~500MB free RAM to load 5 million mails.
>
> By the time Mark stopped working on UW IMAP (the reference IMAP
> implementation, aka Panda IMAP), c.2010, even top-of-the-range
> smartphones typically had only ~512MB RAM total, and top-of-the-range
> ThinkPads had max ~4GiB (which was the upper limit of what 32-bit
> operating systems, still popular then, could handle).
>
> When IMAP2 was invented, c.1988-1990, RAM like that was basically
> supercomputer territory.
>
> Had Mark intended or expected IMAP2 users to have had supercomputers at
> their disposal, and to be searching such large volumes of mail over the
> protocol, I suspect he would have designed the protocol differently: for
> raw efficiency over human readability.
>
> My advice for anyone who wants to be able to keep and search very large
> monolithic mailstores would be: synchronise (e.g. via
> https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison
> or whatever) those mailstores from the server onto your local
> filesystem; use maildir on your local filesystem; and use either Mutt's
> "limiting" functions, or notmuch's index/search functions, for
> searching/browsing.
>
> Good luck in your quest!
>
> Sam

Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox

[Turritopsis Dohrnii Teo En Ming]

I believe Gmail is using IMAP. The instructions for configuring Gmail
email accounts in Outlook specifically mention IMAP server hostname:

imap.gmail.com

TCP Port 993, SSL

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
20 Apr 2022 Wednesday


On Wed, 20 Apr 2022 at 09:26, Shawn Heisey  wrote:
>
> On 4/19/2022 5:12 PM, Turritopsis Dohrnii Teo En Ming wrote:
> > It is not true that no email system was ever designed for 200,000 or
> > more emails.
> >
> > If I did not remember wrongly, more than 10 years ago, I used to have
> > a Gmail email account, with 4-5 MILLION email messages in the Inbox.
> > There was no problem searching even though I had 4-5 MILLION email
> > messages in the Inbox.
>
> If you were using either a Google app or the gmail.com website to access
> that inbox, then it doesn't compare.  I really doubt that either of
> those uses the IMAP protocol.  They would be using something proprietary
> that is highly optimized for the way that Google stores data and
> leverages the enormous amounts of computing power that they maintain.
>
> I would bet that if you accessed a gmail folder with 5 million messages
> in it using IMAP, you would have similar problems with it to those that
> have been described here in this thread.  IMAP is a beautiful protocol,
> but I don't think it was designed for handling that many messages.
>
> Thanks,
> Shawn
>

Re: temporary block incoming messages to specific user

[justina colmena ~biz]

So the file "/var/mail/username" is a "system inbox" for the user, typically a 
flat file that will accept new mail no matter what as long as it isn't too 
large, which would indicate that the user's mailbox is full.

Some of the early text clients, mutt etc. would move any mail in the 
"/var/mail/username" inbox to a local inbox in "/home/username/Maildir" or 
"/home/username/mbox" as soon as it is seen or read from the system inbox.

Are you blocking the user from logging in, or do you just want incoming mail 
for that user to sit in the general queue until you have that person's account 
set up?

IMHO dovecot or other clients should ideally pick up any mail in the system 
inbox "/var/mail/username" and move it to a local maildir inbox in the user's 
home folder as expeditiously as possible for any further reading or sorting.

On April 20, 2022 4:39:24 AM AKDT, Marc  wrote:
>Is it possible to block incoming messages from being delivered to a specific 
>user in such a way that the MTA will try again later. I do not want these 
>message to bounce. (eg while doing some manual maintenance on the user)?
>
>
>
>
>
>
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: temporary block incoming messages to specific user

[Paul Kudla (SCOM.CA Internet Services Inc.)]

this is handled in a couple of different ways


please note this is only for virtual users and assumes you are running 
postfix/dovecot



through postfix (first)

see main.cf :

# SOFT BOUNCE
#
# The soft_bounce parameter provides a limited safety net for
# testing.  When soft_bounce is enabled, mail will remain queued that
# would otherwise bounce. This parameter disables locally-generated
# bounces, and prevents the SMTP server from rejecting mail permanently
# (by changing 5xx replies into 4xx replies). However, soft_bounce
# is no cure for address rewriting mistakes or mail routing mistakes.
#

soft_bounce = yes


the above option in postfix will continually try to deliver to the end 
user, if the user does not exist it will continue trying without any 
bounceback to the sender.


this was built into postfix mainly for mail migrations between servers 
where the dovecot, cyrus server upgrades etc came into play


but should work for your situation.


next on the dovecot side (user database queries)

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql 
password=Scom411400

default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'


user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


#iterate_query = SELECT user, password FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False



this file reflects how to query the database tables etc

note the status=true above etc, i use this to specify if the account is 
active and ok or not


if this flag is set to false, dovecot will consider this user non active 
and not allow delivery locally.


mainly used for billing but will sufice

also note i use an alias flag (from the cyrus days) to forward emails 
off to another actual active account


i am still streamlining that internally but thought i should explain why 
it is there


please note iterate_query is used for the replication users (doveadm 
user '*') query - not needed if you dont run replication but wont hurt 
anything if it is there.










Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/20/2022 8:39 AM, Marc wrote:

Is it possible to block incoming messages from being delivered to a specific 
user in such a way that the MTA will try again later. I do not want these 
message to bounce. (eg while doing some manual maintenance on the user)?

temporary block incoming messages to specific user

[Marc]

Is it possible to block incoming messages from being delivered to a specific 
user in such a way that the MTA will try again later. I do not want these 
message to bounce. (eg while doing some manual maintenance on the user)?

RE: disabling namespace in special-userdb on dovecot 2.2

[Marc]

> 
> Currently I have such special-userdb file
> 
> test:x:1:2:testaccount_descr:/home/users/testaccount:/bin/false:userdb_
> mail=mbox:~/mbox:INBOX=/home/users/testaccount/inbox:INDEX=/home/users/testacco
> unt/index
> 
> However I am still getting errors of a default configured namespace that still
> seems to be active. Is there a way to disable this namespace or reconfigure
> this in the userdb file? (When I was testing this on a dovecot 2.3 I did not
> run into this)
> 

userdb_mail_debug=yes userdb_namespace/archives/disabled=yes