Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Guillaume Barrette

Hi Bill,

  Thanks again for your help!

Regarding the rules that apply tags, actually I had one for each emails 
that added a tag to each new email and that were changed at a later time 
by a Google Apps Script. I now removed them and removed the script. I'll 
see if this changes something (actually I did it some hours ago and with 
only 1 email online in MM and I still see the download raise a little 
too much; seems to be less than before, but still a lot more than what 
Sam gets).


Now I've been running with my biggest email account (102,819 emails) for 
about 2 or maybe 3 hours and I have 170 mb downloads. So, I guess this 
is the one that is the most problematic... I'll keep looking at it.


You said that Gmail auto tagging could cause the download of email 
multiple times, could this happen with the auto mark as Important in 
Gmail ? and when we flag an email (starred) ? Or since those 2 mailboxes 
are not subscribed, then this shouldn't cause any problem?


Thanks,

--
Guillaume

On 10 May 2020, at 22:21, Bill Cole wrote:


On 10 May 2020, at 21:43, Guillaume Barrette wrote:


Hi Sam,

   Thanks a lot for doing the test, this is really appreciated and 
give me a better cue that something shouldn't be right with my setup!


I was thinking that maybe the problem was with Gmail since my 3 
emails who are really used are from Gmail, the 2 others are from my 
web hosting provider and only have 109 messages in one and 21 in the 
other, but now that you said that you have 2 Gmails addresses, then 
there must be something else.


GMail does seem like a potential source of heavy traffic, since they 
conflate IMAP flags (which MM exposes as "Tags") with mailboxes, so 
that their mailboxes are a bit like "Smart" mailboxes. If you have 
rules that tag mail automatically you could end up downloading mail 
twice or more.



So, do you mind giving me some insight on how your Gmail accounts are 
configured in MailMate? Myself, I followed the information that we 
can find in MailMate manual 
[here](https://manual.mailmate-app.com/account_setup). So my 
`Important` + `Starred` mailboxes are unchecked (not subscribed), but 
I kept checked my `All Mail` mailbox (subscribed). Is it what you 
have or you also unchecked the `All Mail` mailbox ?


I also have 2 GMail accounts, one with the recommended subscriptions 
and one with all mailboxes subscribed. I do not see heavy bandwidth 
usage, but I also have very little mail activity on those accounts.


Now I think I'll put my email accounts offline and activate one for a 
while and do this procedure with each account and see if one is the 
problem or if it's MailMate in general.


An excellent approach.

One thing I have seen (and submitted a bug report for years ago) is 
that MM can get into an infinite loop state where it repeatedly runs 
the same IMAP commands against the same mailbox until the mailbox is 
offlined. This could cause heavy bandwidth use but you would probably 
notice that because it comes with an activity spinner for the account 
& mailbox stricken by the bug.



One question for anyone reading, I don't think this is the case since 
from my understanding this should simply scan the local copy of the 
emails, but does the more that we have smart mailboxes in MailMate 
the more it needs to fetch emails online?


No. Unless you have rules attached to those smart mailboxes that tag 
or move messages automatically, they should not cause any server 
interactions.




--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Bill Cole

On 10 May 2020, at 21:43, Guillaume Barrette wrote:


Hi Sam,

   Thanks a lot for doing the test, this is really appreciated and 
give me a better cue that something shouldn't be right with my setup!


I was thinking that maybe the problem was with Gmail since my 3 emails 
who are really used are from Gmail, the 2 others are from my web 
hosting provider and only have 109 messages in one and 21 in the 
other, but now that you said that you have 2 Gmails addresses, then 
there must be something else.


GMail does seem like a potential source of heavy traffic, since they 
conflate IMAP flags (which MM exposes as "Tags") with mailboxes, so that 
their mailboxes are a bit like "Smart" mailboxes. If you have rules that 
tag mail automatically you could end up downloading mail twice or more.



So, do you mind giving me some insight on how your Gmail accounts are 
configured in MailMate? Myself, I followed the information that we can 
find in MailMate manual 
[here](https://manual.mailmate-app.com/account_setup). So my 
`Important` + `Starred` mailboxes are unchecked (not subscribed), but 
I kept checked my `All Mail` mailbox (subscribed). Is it what you have 
or you also unchecked the `All Mail` mailbox ?


I also have 2 GMail accounts, one with the recommended subscriptions and 
one with all mailboxes subscribed. I do not see heavy bandwidth usage, 
but I also have very little mail activity on those accounts.


Now I think I'll put my email accounts offline and activate one for a 
while and do this procedure with each account and see if one is the 
problem or if it's MailMate in general.


An excellent approach.

One thing I have seen (and submitted a bug report for years ago) is that 
MM can get into an infinite loop state where it repeatedly runs the same 
IMAP commands against the same mailbox until the mailbox is offlined. 
This could cause heavy bandwidth use but you would probably notice that 
because it comes with an activity spinner for the account & mailbox 
stricken by the bug.



One question for anyone reading, I don't think this is the case since 
from my understanding this should simply scan the local copy of the 
emails, but does the more that we have smart mailboxes in MailMate the 
more it needs to fetch emails online?


No. Unless you have rules attached to those smart mailboxes that tag or 
move messages automatically, they should not cause any server 
interactions.




--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Guillaume Barrette

Hi Sam,

   Thanks a lot for doing the test, this is really appreciated and 
give me a better cue that something shouldn't be right with my setup!


I was thinking that maybe the problem was with Gmail since my 3 emails 
who are really used are from Gmail, the 2 others are from my web hosting 
provider and only have 109 messages in one and 21 in the other, but now 
that you said that you have 2 Gmails addresses, then there must be 
something else.


So, do you mind giving me some insight on how your Gmail accounts are 
configured in MailMate? Myself, I followed the information that we can 
find in MailMate manual 
[here](https://manual.mailmate-app.com/account_setup). So my `Important` 
+ `Starred` mailboxes are unchecked (not subscribed), but I kept checked 
my `All Mail` mailbox (subscribed). Is it what you have or you also 
unchecked the `All Mail` mailbox ?


Now I think I'll put my email accounts offline and activate one for a 
while and do this procedure with each account and see if one is the 
problem or if it's MailMate in general.


One question for anyone reading, I don't think this is the case since 
from my understanding this should simply scan the local copy of the 
emails, but does the more that we have smart mailboxes in MailMate the 
more it needs to fetch emails online?


Thanks again Sam and everyone!

--
Guillaume

On 10 May 2020, at 20:53, Sam Hathaway wrote:

Just a follow up - having run MailMate for about 18 hours with 
Activity Monitor open, I’m showing 1 MB sent and 20 MB received.


Not a huge email day for me, but that’s like three orders of 
magnitude less than what you’re seeing. I think there’s something 
screwy going on with your setup.


Maybe some brain-dead IMAP servers force MailMate to be more 
profligate with bandwidth. What servers are you connecting to? I have 
two Fastmail accounts, two GMail accounts, and one private server 
running Dovecot.


Hope this helps and that you find a fix.
-sam

On 10 May 2020, at 12:25, Guillaume Barrette wrote:


Hi Bill,

Actually, if you open Activity Monitor, you will see it 
starts at zero and then augment and if you restart it it restarts. 
With that, I'm watching it during the day and see that the download 
count raise little by little. For example, I restarted Activity 
Monitor this morning (a little before I sent my last email) and 
without restarting MailMate and now I see 522 ko uploads / 325,4 mo 
uploads [screenshot](https://d.pr/i/6iuelW)


So, yes at first I thought this could have been a total since the 
process started, but I see while looking at it that it rises and 
restart on restart (or pane change) of Activity Monitor, so I'm quite 
sure this is accumulated between each time I restart it.


Thanks for your inputs, I'll try to see if I can find another place 
where I could see the total since the process started to see the 
difference.


Best,

--
Guillaume

On 10 May 2020, at 12:16, Bill Cole wrote:


On 10 May 2020, at 10:32, Guillaume Barrette wrote:

To give you more inputs, yesterday MailMate downloaded for 2.1gb of 
data.
You can see a screenshot taken at the end of the day before I put 
my laptop to sleep [here](https://d.pr/i/XVsY6a)


I believe those network values are not per day, but for the lifetime 
of the process. I haven't found (in a brief search) any explicit 
documentation of it being anything else. Sleep (even hibernation) 
does not reset the counter.


--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate




___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate




___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Sam Hathaway
Just a follow up - having run MailMate for about 18 hours with Activity 
Monitor open, I’m showing 1 MB sent and 20 MB received.


Not a huge email day for me, but that’s like three orders of magnitude 
less than what you’re seeing. I think there’s something screwy going 
on with your setup.


Maybe some brain-dead IMAP servers force MailMate to be more profligate 
with bandwidth. What servers are you connecting to? I have two Fastmail 
accounts, two GMail accounts, and one private server running Dovecot.


Hope this helps and that you find a fix.
-sam

On 10 May 2020, at 12:25, Guillaume Barrette wrote:


Hi Bill,

Actually, if you open Activity Monitor, you will see it starts 
at zero and then augment and if you restart it it restarts. With that, 
I'm watching it during the day and see that the download count raise 
little by little. For example, I restarted Activity Monitor this 
morning (a little before I sent my last email) and without restarting 
MailMate and now I see 522 ko uploads / 325,4 mo uploads 
[screenshot](https://d.pr/i/6iuelW)


So, yes at first I thought this could have been a total since the 
process started, but I see while looking at it that it rises and 
restart on restart (or pane change) of Activity Monitor, so I'm quite 
sure this is accumulated between each time I restart it.


Thanks for your inputs, I'll try to see if I can find another place 
where I could see the total since the process started to see the 
difference.


Best,

--
Guillaume

On 10 May 2020, at 12:16, Bill Cole wrote:


On 10 May 2020, at 10:32, Guillaume Barrette wrote:

To give you more inputs, yesterday MailMate downloaded for 2.1gb of 
data.
You can see a screenshot taken at the end of the day before I put my 
laptop to sleep [here](https://d.pr/i/XVsY6a)


I believe those network values are not per day, but for the lifetime 
of the process. I haven't found (in a brief search) any explicit 
documentation of it being anything else. Sleep (even hibernation) 
does not reset the counter.


--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate




___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Guillaume Barrette

Hi Bill,

Actually, if you open Activity Monitor, you will see it starts 
at zero and then augment and if you restart it it restarts. With that, 
I'm watching it during the day and see that the download count raise 
little by little. For example, I restarted Activity Monitor this morning 
(a little before I sent my last email) and without restarting MailMate 
and now I see 522 ko uploads / 325,4 mo uploads 
[screenshot](https://d.pr/i/6iuelW)


So, yes at first I thought this could have been a total since the 
process started, but I see while looking at it that it rises and restart 
on restart (or pane change) of Activity Monitor, so I'm quite sure this 
is accumulated between each time I restart it.


Thanks for your inputs, I'll try to see if I can find another place 
where I could see the total since the process started to see the 
difference.


Best,

--
Guillaume

On 10 May 2020, at 12:16, Bill Cole wrote:


On 10 May 2020, at 10:32, Guillaume Barrette wrote:

To give you more inputs, yesterday MailMate downloaded for 2.1gb of 
data.
You can see a screenshot taken at the end of the day before I put my 
laptop to sleep [here](https://d.pr/i/XVsY6a)


I believe those network values are not per day, but for the lifetime 
of the process. I haven't found (in a brief search) any explicit 
documentation of it being anything else. Sleep (even hibernation) does 
not reset the counter.


--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Bill Cole

On 10 May 2020, at 10:32, Guillaume Barrette wrote:

To give you more inputs, yesterday MailMate downloaded for 2.1gb of 
data.
You can see a screenshot taken at the end of the day before I put my 
laptop to sleep [here](https://d.pr/i/XVsY6a)


I believe those network values are not per day, but for the lifetime of 
the process. I haven't found (in a brief search) any explicit 
documentation of it being anything else. Sleep (even hibernation) does 
not reset the counter.


--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-10 Thread Guillaume Barrette

Hi all!

First, thanks Travis, Bill and Randall for your insight on how IMAP 
works. I think that would have been long and demanding to scan every 
email when syncing, so I'm glad this is not the case


Thanks Patrik for your workflow regarding your procedure to archive your 
emails, I'll keep this as well for when I need to move my emails out of 
MM.


As I said, at the moment I can live with the disk space + RAM + MailMate 
general performance while working with it since I don't find it that 
slow. I know it would be snappier while searching and doing operations, 
but I think I'll investigate a little more to see if I can find the 
cause of the high download bandwidth.


To give you more inputs, yesterday MailMate downloaded for 2.1gb of 
data.
You can see a screenshot taken at the end of the day before I put my 
laptop to sleep [here](https://d.pr/i/XVsY6a)


Regarding the RAM (I think it's simply because of the number of emails, 
but if anyone find something that doesn't look right regarding the RAM 
or anything else, please let me know!);


- [Screenshot from Activity Monitor](https://d.pr/i/UEw5dy)

- From the "top" terminal command:

```
PIDCOMMAND  %CPU TIME #TH   #WQ  #PORT MEMPURG   CMPRS  
PGRP  PPID  STATEBOOSTS   %CPU_ME %CPU_OTHRS UID  FAULTS
COW  MSGSENTMSGRECVSYSBSD SYSMACHCSW PAGEINS 
 IDLEW  POWE
24238  MailMate 1.4  77:01.88 2527071  1164M  2088K  736M   
24238 1 sleeping *0[5479] 0.00236 0.0501  9284220   
3535 10108349+  2441046+   66702119+  38571993+  31311669+   294676  
 698677+1.6

```

- From the Instruments application:

```
Process ID	Process Name		Responsible Process	User Name			% CPU	CPU Time	 
# Threads	Memory	  Kind			Sudden Termination	Sandbox	Restricted	App 
Nap	Idle Wake Ups	Disk Writes (B)	Disk Reads (B)	
24238		 MailMate (24238)	n/a	   		 GuillaumeBarrette	2.1%	 77.11 
min	26		   1.13 GiB	Intel (64 bit)	No	No			No		No		1,739,433		639.85 
MiB		 2.40 GiB	

```

So, I'll keep investigating the bandwidth and see if I find a solution 
(or do you think I should fill a bug report?)


Thanks again everyone for your inputs, this is really appreciated!

Best,

--
Guillaume

On 10 May 2020, at 1:07, Patrik Fältström via mailmate wrote:


On 9 May 2020, at 16:45, Guillaume Barrette wrote:

So, I think my best way to reduce most of it would be to move a lot 
of my emails to a new submailbox that I will unsubscribe from 
MailMate. So my question, is this the only solution?


FWIW, your usage is MUCH smaller than mine. If I had only your 
situation, I would not start doing what you are doing...  :-) :-) :-)


That said, your plan is exactly what I have implemented.

What I have is a virtual mailbox ("To Archive") which include all 
email that is older than one year, with sub mailboxes which is "one 
month per sub mailbox".


Then I move all mail which is in these "To Archive" sub mailboxes to 
mailboxes on this separate account which only have this older mail.


Two important things:

- The actual move, the archiving, is something I do manually

- I do *NOT* move mail that I have flags on, those stay

Implement as follows:

1. Create a virtual mailbox "The Mail" that includes all mail that is 
to be archived, for example the following mailboxes. No rules for this 
mailbox. This is btw the one you base all actions on (all other 
mailboxes) instead of "All Mail". For example "Unread", "Default 
mailbox for searches" etc.


1.1. Mail NOT in the archive itself (in that account)
1.2. Mail NOT in Draft
1.3. Mail NOT in Junk
1.4. Mail NOT in Deleted Messages

2. Create the mailbox "To Archive" with the following characteristics 
(2.2.3 is to not have data for the Apple Notes application be moved 
away):


2.1. Act on "The Mail" as the mailbox
2.2. Conditions, where all of the following are true:
2.2.1. "Date is not within 1 year"
2.2.2. "Message is not flagged"
2.2.3. "X-Uniform-Type-Identifier" is not com.apple.mail-note
2.3. For sub mailboxes set:
2.3.1. Have a sub mailbox for each unique value of Date->Month
2.3.2. Set name of sub mailbox to "${#date.month}"

Then you create one folder in your archive account which have the same 
name as the sub mailboxes within "To Archive", open the sub mailbox, 
select all, and just drag the mail over.


This 2nd account, with the archive you then do not have to for example 
add on your phone etc. And more importantly, as you act all Mailmate 
commands and virtual folders on the "The Mail" folder, the speed up 
with be "a lot".


   Patrik

P.S. FWIW, I get 12-13k messages a month, my MailMate process is 5-6G 
and I have a total of 2.8 million email messages by now...adding 
12-15k per month.

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.fr

Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Patrik Fältström via mailmate
On 9 May 2020, at 16:45, Guillaume Barrette wrote:

> So, I think my best way to reduce most of it would be to move a lot of my 
> emails to a new submailbox that I will unsubscribe from MailMate. So my 
> question, is this the only solution?

FWIW, your usage is MUCH smaller than mine. If I had only your situation, I 
would not start doing what you are doing...  :-) :-) :-)

That said, your plan is exactly what I have implemented.

What I have is a virtual mailbox ("To Archive") which include all email that is 
older than one year, with sub mailboxes which is "one month per sub mailbox".

Then I move all mail which is in these "To Archive" sub mailboxes to mailboxes 
on this separate account which only have this older mail.

Two important things:

- The actual move, the archiving, is something I do manually

- I do *NOT* move mail that I have flags on, those stay

Implement as follows:

1. Create a virtual mailbox "The Mail" that includes all mail that is to be 
archived, for example the following mailboxes. No rules for this mailbox. This 
is btw the one you base all actions on (all other mailboxes) instead of "All 
Mail". For example "Unread", "Default mailbox for searches" etc.

1.1. Mail NOT in the archive itself (in that account)
1.2. Mail NOT in Draft
1.3. Mail NOT in Junk
1.4. Mail NOT in Deleted Messages

2. Create the mailbox "To Archive" with the following characteristics (2.2.3 is 
to not have data for the Apple Notes application be moved away):

2.1. Act on "The Mail" as the mailbox
2.2. Conditions, where all of the following are true:
2.2.1. "Date is not within 1 year"
2.2.2. "Message is not flagged"
2.2.3. "X-Uniform-Type-Identifier" is not com.apple.mail-note
2.3. For sub mailboxes set:
2.3.1. Have a sub mailbox for each unique value of Date->Month
2.3.2. Set name of sub mailbox to "${#date.month}"

Then you create one folder in your archive account which have the same name as 
the sub mailboxes within "To Archive", open the sub mailbox, select all, and 
just drag the mail over.

This 2nd account, with the archive you then do not have to for example add on 
your phone etc. And more importantly, as you act all Mailmate commands and 
virtual folders on the "The Mail" folder, the speed up with be "a lot".

   Patrik

P.S. FWIW, I get 12-13k messages a month, my MailMate process is 5-6G and I 
have a total of 2.8 million email messages by now...adding 12-15k per month.


signature.asc
Description: OpenPGP digital signature
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Randall Meadows

On 9 May 2020, at 16:32, Bill Cole wrote:


On 9 May 2020, at 15:45, Travis Risner wrote:


Hi Guillaume,

I have no direct knowledge of how MM works, but I have suspicions.  
If MM is __synchronizing__ with all the IMAP servers, then I suspect 
that it has to do something for each of the 200,000 (or 500,000) 
emails to verify that it is still there and hasn’t changed.


One fundamental rule of IMAP is that messages never change. Message 
flags can change, but the combination of mailbox name, UIDVALIDITY 
value for the mailbox, and UID must refer to a single immutable 
message on that server forever. Re-downloading a message only happens 
if the client or server has a major failure that forces a rebuild of 
its message indices.


Also (and please correct me if I'm wrong), if you store messages in 
mailboxes other than INBOX (like, an Archive or some such), and you've 
not subscribed to that mailbox, MM will never even look at it, during 
its normal duties, until you move something in there, right?

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Bill Cole

On 9 May 2020, at 15:45, Travis Risner wrote:


Hi Guillaume,

I have no direct knowledge of how MM works, but I have suspicions.  If 
MM is __synchronizing__ with all the IMAP servers, then I suspect that 
it has to do something for each of the 200,000 (or 500,000) emails to 
verify that it is still there and hasn’t changed.


One fundamental rule of IMAP is that messages never change. Message 
flags can change, but the combination of mailbox name, UIDVALIDITY value 
for the mailbox, and UID must refer to a single immutable message on 
that server forever. Re-downloading a message only happens if the client 
or server has a major failure that forces a rebuild of its message 
indices.


Maybe it downloads each email and compares checksums.  Maybe it has a 
clever way of interacting with the IMAP server so it does not have to 
download every byte of each email.  Even so, MM still has to do 
something for each email every time it checks.  That might be what is 
chewing up all the bandwidth.


IMAP synchronization does not usually require examination of every 
message. Instead, when a SELECT command is executed for a mailbox, the 
server responds with mailbox metadata which is roughly 400 bytes total, 
describing the state of the mailbox and its messages. This allows the 
client to determine what it needs to do to synchronize the mailbox. If 
nothing in a mailbox has changed since last sync and the server supports 
the "CONDSTORE" extension (as most do,) the client doesn't even need to 
look at any message flags. Without CONDSTORE, reliable synchronization 
requires the client to check the flags of all messages in a mailbox, 
which is a few dozen bytes per message in extreme cases.



--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not For Hire (currently)
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Travis Risner

Hi Guillaume,

I have no direct knowledge of how MM works, but I have suspicions.  If 
MM is __synchronizing__ with all the IMAP servers, then I suspect that 
it has to do something for each of the 200,000 (or 500,000) emails to 
verify that it is still there and hasn’t changed.  Maybe it downloads 
each email and compares checksums.  Maybe it has a clever way of 
interacting with the IMAP server so it does not have to download every 
byte of each email.  Even so, MM still has to do something for each 
email every time it checks.  That might be what is chewing up all the 
bandwidth.


My solution is similar to Tracy’s.  I run MM on my laptop and run a 
different email program (Thunderbird) on my “server” at home.  That 
email program has rules that will transfer selected emails (that I 
don’t need to see immediately) to local email folders and remove those 
emails from the IMAP server.  For other emails that I have looked at but 
don’t need online any more, I manually transfer to other local email 
folders which also removes them from the IMAP server.  Thus, the only 
emails I keep on the IMAP servers are the ones that I want to do 
something about later.


I too will be looking at Mail Steward and Horcrux to see if that is a 
better solution.


HTH,

Travis

--
Travis Risner

On 5/9/20 2:46 PM, Guillaume Barrette wrote:

Dear Tracy,

   Thanks for your reply, this is greatly appreciated!

Yes, I understand around 200,000 emails may be quite a lot and no I 
don't need all of them instantly and yes I was thinking of doing 
something to reduce that (backup + moving emails to submailboxes that 
would be unsubscribed). However, I saw some other mentions of people 
having 200,000 or 250,000 or even 500,000 emails in MailMate, so I was 
wondering if my count was that high before making the move.


On my side, I don't find MailMate really slow with that amount of 
emails, it's really regarding the bandwidth + disk space + RAM. I know 
the disk space + RAM would shrink by having fewer emails, but was 
wondering if the bandwidth would be the same since the only emails 
that are touched are the recent ones, so does removing the old ones 
will really reduce the bandwidth or it is simply how it works and if 
so I'll need to find a way around (maybe raising the delay for the 
Synchronization Schedule of most mailboxes will help or creating a 
script to toggle the Online/Offline state of the mailboxes could 
help...)


With other email clients I wasn't synchronizing all those years for 
all my email accounts since there was a feature to only synchronize X 
months, but I like MailMate and I'm sure I'll find a way to put things 
in a shape that I like, but I was wondering regarding the different 
options that I could approach those points before making a big move.


With that said, thanks for giving an insight on your workflow and 
mentioning MailSteward and Horcrux (I didn't know this last one), I 
may go for one of them.


Thanks for your help,

--
Guillaume

On 9 May 2020, at 13:31, Tracy Valleau wrote:


Hello,

Do you really need 200,000 emails to be instantly available to you, 
or are you using 99% of that just to store old emails?


If the latter, then you could inprove your situation immensely by 
using an email archiver, such as Horcrux, or (my preferred) 
MailSteward (which I have been using for well over a decade.)


These will move your emails into a database, and allow you to remove 
them from your server (ie: delete them).


I filter out my spam first, and then use MailSteward to archive the 
rest. It has never failed me, and is quite fast at finding emails. My 
collection goes back to 1993.


You can use either the SQLite version, or the MySQL version. I used 
MySQL for a very long time, and then realized that I really never 
referred to 20-year old emails, so I put them into a SQLite table, 
and saved it out separately. I can load it in if I ever need to find 
an old email, and now I keep only the past 6 or so years active in 
MailSteward at any one time. My own copy of MailMail usually runs a 
total < 50 active emails, so it is lightning fast, and resource 
light. If I really need to see an old email, I just run MailSteward 
and look it up. In practice, I may do that twice a week or so.


Note that this works because while I have probably 30 email addresses 
on my server (I'm a developer), I have them all forward to a single 
mailbox. I did that originally because that way I only needed one 
email account in MailMate (the account everything is forwarded to) 
and MM can still filter everything based on the original address a 
given email was sent to. MUCH more efficient than checking 30 
different accounts from my computer!


AND, that in turn allows me to use MailSteward, which is designed for 
Apple Mail, but conveniently has a "also collect emails from this 
folder" which I have filled with my local storage: 
"file:///Users/tracyv/Library/Application%20Support/MailMate/Messages/IMAP/tracy%2540mymail...

Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Sam Hathaway
2 GB of network traffic per day seems high. I’m going to leave 
Activity monitor open on my MailMate today and see what I get.


Messages.noindex tends to be a bit larger than the on-server size of the 
messages, but not staggeringly so. For example, one of my accounts is 
using 4.7 GB on the server but its folder inside Messages.noindex is 5.0 
GB.


Also I’d suggest paying attention to the “Private memory” rather 
than “Memory” as that shows the amount of RAM MailMate is using that 
isn’t shared with other processes. (If I understand correctly.)


Hope this helps.
-sam

On 9 May 2020, at 10:45, Guillaume Barrette wrote:


Dear MailMate Users,

First, I want to say that I really like MailMate with its 
customization and search possibilities, this is not a rant, but just 
that I would like to tune it to reduce its footprint if possible.


In short I find it a little heavy regarding its usage of resources. To 
give some statistics:


My mailboxes consist of:

- 195,766 messages collected from 5 mailboxes (3 from Gmail + 2 other 
from my web host which have pretty much no message in them)

- I'm getting around 80 emails per day

What I would like to improve:

1. By investigating MailMate's downloads from the Activity Monitor 
Network tab, I see that MailMate is downloading around 2gb of data per 
days
2. My "Messages.noindex" directory takes around 20gb of space on my 
drive

3. MailMate takes around 1gb of memory while running

So, I think my best way to reduce most of it would be to move a lot of 
my emails to a new submailbox that I will unsubscribe from MailMate. 
So my question, is this the only solution? I know there's no way to 
set MailMate to only download emails from the "last 6 months" or 
something like that, but is there other solutions than to move 
everything out?


Regarding my MailMate download statistic, my Internet plan consists of 
150gb per month, so almost half of it goes to MailMate (2gb x 30 days 
= 60gb / 150gb), is this normal our could it be a bug? I don't think 
my previous email application was using that much bandwidth... I have 
difficulty to see how MailMate can use that much bandwidth since by 
looking at a big day (105 mails) it's around 6.8 mb of storage space. 
Is MailMate batch re-downloading many emails every day ? To mention, 
all my accounts have their "Synchronization Schedule" set to "Every 10 
minutes".


I have multiple Smart Mailboxes, with some submailboxes populated 
using the "Submailbox for each unique value of", could this add to the 
bandwidth and does reducing the amount will greatly help in reducing 
the memory used or the memory is more proportional to the number of 
emails?


Thanks anyone for any cues and help!

All the best,

--
Guillaume
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Guillaume Barrette

Dear Tracy,

  Thanks for your reply, this is greatly appreciated!

Yes, I understand around 200,000 emails may be quite a lot and no I 
don't need all of them instantly and yes I was thinking of doing 
something to reduce that (backup + moving emails to submailboxes that 
would be unsubscribed). However, I saw some other mentions of people 
having 200,000 or 250,000 or even 500,000 emails in MailMate, so I was 
wondering if my count was that high before making the move.


On my side, I don't find MailMate really slow with that amount of 
emails, it's really regarding the bandwidth + disk space + RAM. I know 
the disk space + RAM would shrink by having fewer emails, but was 
wondering if the bandwidth would be the same since the only emails that 
are touched are the recent ones, so does removing the old ones will 
really reduce the bandwidth or it is simply how it works and if so I'll 
need to find a way around (maybe raising the delay for the 
Synchronization Schedule of most mailboxes will help or creating a 
script to toggle the Online/Offline state of the mailboxes could 
help...)


With other email clients I wasn't synchronizing all those years for all 
my email accounts since there was a feature to only synchronize X 
months, but I like MailMate and I'm sure I'll find a way to put things 
in a shape that I like, but I was wondering regarding the different 
options that I could approach those points before making a big move.


With that said, thanks for giving an insight on your workflow and 
mentioning MailSteward and Horcrux (I didn't know this last one), I may 
go for one of them.


Thanks for your help,

--
Guillaume

On 9 May 2020, at 13:31, Tracy Valleau wrote:


Hello,

Do you really need 200,000 emails to be instantly available to you, or 
are you using 99% of that just to store old emails?


If the latter, then you could inprove your situation immensely by 
using an email archiver, such as Horcrux, or (my preferred) 
MailSteward (which I have been using for well over a decade.)


These will move your emails into a database, and allow you to remove 
them from your server (ie: delete them).


I filter out my spam first, and then use MailSteward to archive the 
rest. It has never failed me, and is quite fast at finding emails. My 
collection goes back to 1993.


You can use either the SQLite version, or the MySQL version. I used 
MySQL for a very long time, and then realized that I really never 
referred to 20-year old emails, so I put them into a SQLite table, and 
saved it out separately. I can load it in if I ever need to find an 
old email, and now I keep only the past 6 or so years active in 
MailSteward at any one time. My own copy of MailMail usually runs a 
total < 50 active emails, so it is lightning fast, and resource light. 
If I really need to see an old email, I just run MailSteward and look 
it up. In practice, I may do that twice a week or so.


Note that this works because while I have probably 30 email addresses 
on my server (I'm a developer), I have them all forward to a single 
mailbox. I did that originally because that way I only needed one 
email account in MailMate (the account everything is forwarded to) and 
MM can still filter everything based on the original address a given 
email was sent to. MUCH more efficient than checking 30 different 
accounts from my computer!


AND, that in turn allows me to use MailSteward, which is designed for 
Apple Mail, but conveniently has a "also collect emails from this 
folder" which I have filled with my local storage: 
"file:///Users/tracyv/Library/Application%20Support/MailMate/Messages/IMAP/tracy%2540mymail@mail.mymail.org/INBOX.mailbox/dreamhost.mailbox/"


Upshot? I have every non-spam email I've received for the past 27 
years, and I can find any given one of them almost instantly.


Works for me, but of course YMMV.

HTH


www.valleau.art

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


[MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Tracy Valleau

Hello,

Do you really need 200,000 emails to be instantly available to you, or 
are you using 99% of that just to store old emails?


If the latter, then you could inprove your situation immensely by using 
an email archiver, such as Horcrux, or (my preferred) MailSteward (which 
I have been using for well over a decade.)


These will move your emails into a database, and allow you to remove 
them from your server (ie: delete them).


I filter out my spam first, and then use MailSteward to archive the 
rest. It has never failed me, and is quite fast at finding emails. My 
collection goes back to 1993.


You can use either the SQLite version, or the MySQL version. I used 
MySQL for a very long time, and then realized that I really never 
referred to 20-year old emails, so I put them into a SQLite table, and 
saved it out separately. I can load it in if I ever need to find an old 
email, and now I keep only the past 6 or so years active in MailSteward 
at any one time. My own copy of MailMail usually runs a total < 50 
active emails, so it is lightning fast, and resource light. If I really 
need to see an old email, I just run MailSteward and look it up. In 
practice, I may do that twice a week or so.


Note that this works because while I have probably 30 email addresses on 
my server (I'm a developer), I have them all forward to a single 
mailbox. I did that originally because that way I only needed one email 
account in MailMate (the account everything is forwarded to) and MM can 
still filter everything based on the original address a given email was 
sent to. MUCH more efficient than checking 30 different accounts from my 
computer!


AND, that in turn allows me to use MailSteward, which is designed for 
Apple Mail, but conveniently has a "also collect emails from this 
folder" which I have filled with my local storage: 
"file:///Users/tracyv/Library/Application%20Support/MailMate/Messages/IMAP/tracy%2540mymail@mail.mymail.org/INBOX.mailbox/dreamhost.mailbox/"


Upshot? I have every non-spam email I've received for the past 27 years, 
and I can find any given one of them almost instantly.


Works for me, but of course YMMV.

HTH


www.valleau.art

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Guillaume Barrette
Sorry, I forgot to specify... I'm using the last non-beta version of 
MailMate: Version 1.13.1 (5671)


--
Guillaume

On 9 May 2020, at 10:57, Guillaume Barrette wrote:


Hey Alexandre,

   Let me know what you found out on your side! It would be great 
to see if it's my setup the problem.


To give you some tips, open the Activity Monitor and go into the 
Network Tab. Keep the application open on this state or the bandwidth 
will reset to 0. You can filter to MailMate


Or you can use the Terminal.app and run: nettop
There you can scroll to see MailMate and see its bytes_in for the 
download.


Best,

--
Guillaume

On 9 May 2020, at 10:49, Alexandre Takacs wrote:


On 9 May 2020, at 16:45, Guillaume Barrette wrote:


Thanks anyone for any cues and help!


I'm afraid I don't have anything actionable but I will have a closer 
look to MM and bandwidth usage. I have always felt (and complained) 
that MM was a resource hog but never actually took the time to get 
into taking measurements. Let's do this...



A. Takacs
Augicom SA
+41 (22) 301 16 00
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Guillaume Barrette

Hey Alexandre,

   Let me know what you found out on your side! It would be great to 
see if it's my setup the problem.


To give you some tips, open the Activity Monitor and go into the Network 
Tab. Keep the application open on this state or the bandwidth will reset 
to 0. You can filter to MailMate


Or you can use the Terminal.app and run: nettop
There you can scroll to see MailMate and see its bytes_in for the 
download.


Best,

--
Guillaume

On 9 May 2020, at 10:49, Alexandre Takacs wrote:


On 9 May 2020, at 16:45, Guillaume Barrette wrote:


Thanks anyone for any cues and help!


I'm afraid I don't have anything actionable but I will have a closer 
look to MM and bandwidth usage. I have always felt (and complained) 
that MM was a resource hog but never actually took the time to get 
into taking measurements. Let's do this...



A. Takacs
Augicom SA
+41 (22) 301 16 00
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate

___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


Re: [MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Alexandre Takacs

On 9 May 2020, at 16:45, Guillaume Barrette wrote:


Thanks anyone for any cues and help!


I'm afraid I don't have anything actionable but I will have a closer 
look to MM and bandwidth usage. I have always felt (and complained) that 
MM was a resource hog but never actually took the time to get into 
taking measurements. Let's do this...



A. Takacs
Augicom SA
+41 (22) 301 16 00
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate


[MlMt] Ideas to reduce the footprint of MailMate

2020-05-09 Thread Guillaume Barrette

Dear MailMate Users,

First, I want to say that I really like MailMate with its 
customization and search possibilities, this is not a rant, but just 
that I would like to tune it to reduce its footprint if possible.


In short I find it a little heavy regarding its usage of resources. To 
give some statistics:


My mailboxes consist of:

- 195,766 messages collected from 5 mailboxes (3 from Gmail + 2 other 
from my web host which have pretty much no message in them)

- I'm getting around 80 emails per day

What I would like to improve:

1. By investigating MailMate's downloads from the Activity Monitor 
Network tab, I see that MailMate is downloading around 2gb of data per 
days
2. My "Messages.noindex" directory takes around 20gb of space on my 
drive

3. MailMate takes around 1gb of memory while running

So, I think my best way to reduce most of it would be to move a lot of 
my emails to a new submailbox that I will unsubscribe from MailMate. So 
my question, is this the only solution? I know there's no way to set 
MailMate to only download emails from the "last 6 months" or something 
like that, but is there other solutions than to move everything out?


Regarding my MailMate download statistic, my Internet plan consists of 
150gb per month, so almost half of it goes to MailMate (2gb x 30 days = 
60gb / 150gb), is this normal our could it be a bug? I don't think my 
previous email application was using that much bandwidth... I have 
difficulty to see how MailMate can use that much bandwidth since by 
looking at a big day (105 mails) it's around 6.8 mb of storage space. 
Is MailMate batch re-downloading many emails every day ? To mention, all 
my accounts have their "Synchronization Schedule" set to "Every 10 
minutes".


I have multiple Smart Mailboxes, with some submailboxes populated using 
the "Submailbox for each unique value of", could this add to the 
bandwidth and does reducing the amount will greatly help in reducing the 
memory used or the memory is more proportional to the number of emails?


Thanks anyone for any cues and help!

All the best,

--
Guillaume
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate