Re: [MlMt] Ideas to reduce the footprint of MailMate
Hi Bill, Thanks again for your help! Regarding the rules that apply tags, actually I had one for each emails that added a tag to each new email and that were changed at a later time by a Google Apps Script. I now removed them and removed the script. I'll see if this changes something (actually I did it some hours ago and with only 1 email online in MM and I still see the download raise a little too much; seems to be less than before, but still a lot more than what Sam gets). Now I've been running with my biggest email account (102,819 emails) for about 2 or maybe 3 hours and I have 170 mb downloads. So, I guess this is the one that is the most problematic... I'll keep looking at it. You said that Gmail auto tagging could cause the download of email multiple times, could this happen with the auto mark as Important in Gmail ? and when we flag an email (starred) ? Or since those 2 mailboxes are not subscribed, then this shouldn't cause any problem? Thanks, -- Guillaume On 10 May 2020, at 22:21, Bill Cole wrote: On 10 May 2020, at 21:43, Guillaume Barrette wrote: Hi Sam, Thanks a lot for doing the test, this is really appreciated and give me a better cue that something shouldn't be right with my setup! I was thinking that maybe the problem was with Gmail since my 3 emails who are really used are from Gmail, the 2 others are from my web hosting provider and only have 109 messages in one and 21 in the other, but now that you said that you have 2 Gmails addresses, then there must be something else. GMail does seem like a potential source of heavy traffic, since they conflate IMAP flags (which MM exposes as "Tags") with mailboxes, so that their mailboxes are a bit like "Smart" mailboxes. If you have rules that tag mail automatically you could end up downloading mail twice or more. So, do you mind giving me some insight on how your Gmail accounts are configured in MailMate? Myself, I followed the information that we can find in MailMate manual [here](https://manual.mailmate-app.com/account_setup). So my `Important` + `Starred` mailboxes are unchecked (not subscribed), but I kept checked my `All Mail` mailbox (subscribed). Is it what you have or you also unchecked the `All Mail` mailbox ? I also have 2 GMail accounts, one with the recommended subscriptions and one with all mailboxes subscribed. I do not see heavy bandwidth usage, but I also have very little mail activity on those accounts. Now I think I'll put my email accounts offline and activate one for a while and do this procedure with each account and see if one is the problem or if it's MailMate in general. An excellent approach. One thing I have seen (and submitted a bug report for years ago) is that MM can get into an infinite loop state where it repeatedly runs the same IMAP commands against the same mailbox until the mailbox is offlined. This could cause heavy bandwidth use but you would probably notice that because it comes with an activity spinner for the account & mailbox stricken by the bug. One question for anyone reading, I don't think this is the case since from my understanding this should simply scan the local copy of the emails, but does the more that we have smart mailboxes in MailMate the more it needs to fetch emails online? No. Unless you have rules attached to those smart mailboxes that tag or move messages automatically, they should not cause any server interactions. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
On 10 May 2020, at 21:43, Guillaume Barrette wrote: Hi Sam, Thanks a lot for doing the test, this is really appreciated and give me a better cue that something shouldn't be right with my setup! I was thinking that maybe the problem was with Gmail since my 3 emails who are really used are from Gmail, the 2 others are from my web hosting provider and only have 109 messages in one and 21 in the other, but now that you said that you have 2 Gmails addresses, then there must be something else. GMail does seem like a potential source of heavy traffic, since they conflate IMAP flags (which MM exposes as "Tags") with mailboxes, so that their mailboxes are a bit like "Smart" mailboxes. If you have rules that tag mail automatically you could end up downloading mail twice or more. So, do you mind giving me some insight on how your Gmail accounts are configured in MailMate? Myself, I followed the information that we can find in MailMate manual [here](https://manual.mailmate-app.com/account_setup). So my `Important` + `Starred` mailboxes are unchecked (not subscribed), but I kept checked my `All Mail` mailbox (subscribed). Is it what you have or you also unchecked the `All Mail` mailbox ? I also have 2 GMail accounts, one with the recommended subscriptions and one with all mailboxes subscribed. I do not see heavy bandwidth usage, but I also have very little mail activity on those accounts. Now I think I'll put my email accounts offline and activate one for a while and do this procedure with each account and see if one is the problem or if it's MailMate in general. An excellent approach. One thing I have seen (and submitted a bug report for years ago) is that MM can get into an infinite loop state where it repeatedly runs the same IMAP commands against the same mailbox until the mailbox is offlined. This could cause heavy bandwidth use but you would probably notice that because it comes with an activity spinner for the account & mailbox stricken by the bug. One question for anyone reading, I don't think this is the case since from my understanding this should simply scan the local copy of the emails, but does the more that we have smart mailboxes in MailMate the more it needs to fetch emails online? No. Unless you have rules attached to those smart mailboxes that tag or move messages automatically, they should not cause any server interactions. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Hi Sam, Thanks a lot for doing the test, this is really appreciated and give me a better cue that something shouldn't be right with my setup! I was thinking that maybe the problem was with Gmail since my 3 emails who are really used are from Gmail, the 2 others are from my web hosting provider and only have 109 messages in one and 21 in the other, but now that you said that you have 2 Gmails addresses, then there must be something else. So, do you mind giving me some insight on how your Gmail accounts are configured in MailMate? Myself, I followed the information that we can find in MailMate manual [here](https://manual.mailmate-app.com/account_setup). So my `Important` + `Starred` mailboxes are unchecked (not subscribed), but I kept checked my `All Mail` mailbox (subscribed). Is it what you have or you also unchecked the `All Mail` mailbox ? Now I think I'll put my email accounts offline and activate one for a while and do this procedure with each account and see if one is the problem or if it's MailMate in general. One question for anyone reading, I don't think this is the case since from my understanding this should simply scan the local copy of the emails, but does the more that we have smart mailboxes in MailMate the more it needs to fetch emails online? Thanks again Sam and everyone! -- Guillaume On 10 May 2020, at 20:53, Sam Hathaway wrote: Just a follow up - having run MailMate for about 18 hours with Activity Monitor open, I’m showing 1 MB sent and 20 MB received. Not a huge email day for me, but that’s like three orders of magnitude less than what you’re seeing. I think there’s something screwy going on with your setup. Maybe some brain-dead IMAP servers force MailMate to be more profligate with bandwidth. What servers are you connecting to? I have two Fastmail accounts, two GMail accounts, and one private server running Dovecot. Hope this helps and that you find a fix. -sam On 10 May 2020, at 12:25, Guillaume Barrette wrote: Hi Bill, Actually, if you open Activity Monitor, you will see it starts at zero and then augment and if you restart it it restarts. With that, I'm watching it during the day and see that the download count raise little by little. For example, I restarted Activity Monitor this morning (a little before I sent my last email) and without restarting MailMate and now I see 522 ko uploads / 325,4 mo uploads [screenshot](https://d.pr/i/6iuelW) So, yes at first I thought this could have been a total since the process started, but I see while looking at it that it rises and restart on restart (or pane change) of Activity Monitor, so I'm quite sure this is accumulated between each time I restart it. Thanks for your inputs, I'll try to see if I can find another place where I could see the total since the process started to see the difference. Best, -- Guillaume On 10 May 2020, at 12:16, Bill Cole wrote: On 10 May 2020, at 10:32, Guillaume Barrette wrote: To give you more inputs, yesterday MailMate downloaded for 2.1gb of data. You can see a screenshot taken at the end of the day before I put my laptop to sleep [here](https://d.pr/i/XVsY6a) I believe those network values are not per day, but for the lifetime of the process. I haven't found (in a brief search) any explicit documentation of it being anything else. Sleep (even hibernation) does not reset the counter. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Just a follow up - having run MailMate for about 18 hours with Activity Monitor open, I’m showing 1 MB sent and 20 MB received. Not a huge email day for me, but that’s like three orders of magnitude less than what you’re seeing. I think there’s something screwy going on with your setup. Maybe some brain-dead IMAP servers force MailMate to be more profligate with bandwidth. What servers are you connecting to? I have two Fastmail accounts, two GMail accounts, and one private server running Dovecot. Hope this helps and that you find a fix. -sam On 10 May 2020, at 12:25, Guillaume Barrette wrote: Hi Bill, Actually, if you open Activity Monitor, you will see it starts at zero and then augment and if you restart it it restarts. With that, I'm watching it during the day and see that the download count raise little by little. For example, I restarted Activity Monitor this morning (a little before I sent my last email) and without restarting MailMate and now I see 522 ko uploads / 325,4 mo uploads [screenshot](https://d.pr/i/6iuelW) So, yes at first I thought this could have been a total since the process started, but I see while looking at it that it rises and restart on restart (or pane change) of Activity Monitor, so I'm quite sure this is accumulated between each time I restart it. Thanks for your inputs, I'll try to see if I can find another place where I could see the total since the process started to see the difference. Best, -- Guillaume On 10 May 2020, at 12:16, Bill Cole wrote: On 10 May 2020, at 10:32, Guillaume Barrette wrote: To give you more inputs, yesterday MailMate downloaded for 2.1gb of data. You can see a screenshot taken at the end of the day before I put my laptop to sleep [here](https://d.pr/i/XVsY6a) I believe those network values are not per day, but for the lifetime of the process. I haven't found (in a brief search) any explicit documentation of it being anything else. Sleep (even hibernation) does not reset the counter. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Hi Bill, Actually, if you open Activity Monitor, you will see it starts at zero and then augment and if you restart it it restarts. With that, I'm watching it during the day and see that the download count raise little by little. For example, I restarted Activity Monitor this morning (a little before I sent my last email) and without restarting MailMate and now I see 522 ko uploads / 325,4 mo uploads [screenshot](https://d.pr/i/6iuelW) So, yes at first I thought this could have been a total since the process started, but I see while looking at it that it rises and restart on restart (or pane change) of Activity Monitor, so I'm quite sure this is accumulated between each time I restart it. Thanks for your inputs, I'll try to see if I can find another place where I could see the total since the process started to see the difference. Best, -- Guillaume On 10 May 2020, at 12:16, Bill Cole wrote: On 10 May 2020, at 10:32, Guillaume Barrette wrote: To give you more inputs, yesterday MailMate downloaded for 2.1gb of data. You can see a screenshot taken at the end of the day before I put my laptop to sleep [here](https://d.pr/i/XVsY6a) I believe those network values are not per day, but for the lifetime of the process. I haven't found (in a brief search) any explicit documentation of it being anything else. Sleep (even hibernation) does not reset the counter. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
On 10 May 2020, at 10:32, Guillaume Barrette wrote: To give you more inputs, yesterday MailMate downloaded for 2.1gb of data. You can see a screenshot taken at the end of the day before I put my laptop to sleep [here](https://d.pr/i/XVsY6a) I believe those network values are not per day, but for the lifetime of the process. I haven't found (in a brief search) any explicit documentation of it being anything else. Sleep (even hibernation) does not reset the counter. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Hi all!
First, thanks Travis, Bill and Randall for your insight on how IMAP
works. I think that would have been long and demanding to scan every
email when syncing, so I'm glad this is not the case
Thanks Patrik for your workflow regarding your procedure to archive your
emails, I'll keep this as well for when I need to move my emails out of
MM.
As I said, at the moment I can live with the disk space + RAM + MailMate
general performance while working with it since I don't find it that
slow. I know it would be snappier while searching and doing operations,
but I think I'll investigate a little more to see if I can find the
cause of the high download bandwidth.
To give you more inputs, yesterday MailMate downloaded for 2.1gb of
data.
You can see a screenshot taken at the end of the day before I put my
laptop to sleep [here](https://d.pr/i/XVsY6a)
Regarding the RAM (I think it's simply because of the number of emails,
but if anyone find something that doesn't look right regarding the RAM
or anything else, please let me know!);
- [Screenshot from Activity Monitor](https://d.pr/i/UEw5dy)
- From the "top" terminal command:
```
PIDCOMMAND %CPU TIME #TH #WQ #PORT MEMPURG CMPRS
PGRP PPID STATEBOOSTS %CPU_ME %CPU_OTHRS UID FAULTS
COW MSGSENTMSGRECVSYSBSD SYSMACHCSW PAGEINS
IDLEW POWE
24238 MailMate 1.4 77:01.88 2527071 1164M 2088K 736M
24238 1 sleeping *0[5479] 0.00236 0.0501 9284220
3535 10108349+ 2441046+ 66702119+ 38571993+ 31311669+ 294676
698677+1.6
```
- From the Instruments application:
```
Process ID Process Name Responsible Process User Name % CPU CPU Time
# Threads Memory Kind Sudden Termination Sandbox Restricted App
Nap Idle Wake Ups Disk Writes (B) Disk Reads (B)
24238 MailMate (24238) n/a GuillaumeBarrette 2.1% 77.11
min 26 1.13 GiB Intel (64 bit) No No No No 1,739,433 639.85
MiB 2.40 GiB
```
So, I'll keep investigating the bandwidth and see if I find a solution
(or do you think I should fill a bug report?)
Thanks again everyone for your inputs, this is really appreciated!
Best,
--
Guillaume
On 10 May 2020, at 1:07, Patrik Fältström via mailmate wrote:
On 9 May 2020, at 16:45, Guillaume Barrette wrote:
So, I think my best way to reduce most of it would be to move a lot
of my emails to a new submailbox that I will unsubscribe from
MailMate. So my question, is this the only solution?
FWIW, your usage is MUCH smaller than mine. If I had only your
situation, I would not start doing what you are doing... :-) :-) :-)
That said, your plan is exactly what I have implemented.
What I have is a virtual mailbox ("To Archive") which include all
email that is older than one year, with sub mailboxes which is "one
month per sub mailbox".
Then I move all mail which is in these "To Archive" sub mailboxes to
mailboxes on this separate account which only have this older mail.
Two important things:
- The actual move, the archiving, is something I do manually
- I do *NOT* move mail that I have flags on, those stay
Implement as follows:
1. Create a virtual mailbox "The Mail" that includes all mail that is
to be archived, for example the following mailboxes. No rules for this
mailbox. This is btw the one you base all actions on (all other
mailboxes) instead of "All Mail". For example "Unread", "Default
mailbox for searches" etc.
1.1. Mail NOT in the archive itself (in that account)
1.2. Mail NOT in Draft
1.3. Mail NOT in Junk
1.4. Mail NOT in Deleted Messages
2. Create the mailbox "To Archive" with the following characteristics
(2.2.3 is to not have data for the Apple Notes application be moved
away):
2.1. Act on "The Mail" as the mailbox
2.2. Conditions, where all of the following are true:
2.2.1. "Date is not within 1 year"
2.2.2. "Message is not flagged"
2.2.3. "X-Uniform-Type-Identifier" is not com.apple.mail-note
2.3. For sub mailboxes set:
2.3.1. Have a sub mailbox for each unique value of Date->Month
2.3.2. Set name of sub mailbox to "${#date.month}"
Then you create one folder in your archive account which have the same
name as the sub mailboxes within "To Archive", open the sub mailbox,
select all, and just drag the mail over.
This 2nd account, with the archive you then do not have to for example
add on your phone etc. And more importantly, as you act all Mailmate
commands and virtual folders on the "The Mail" folder, the speed up
with be "a lot".
Patrik
P.S. FWIW, I get 12-13k messages a month, my MailMate process is 5-6G
and I have a total of 2.8 million email messages by now...adding
12-15k per month.
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.fr
Re: [MlMt] Ideas to reduce the footprint of MailMate
On 9 May 2020, at 16:45, Guillaume Barrette wrote:
> So, I think my best way to reduce most of it would be to move a lot of my
> emails to a new submailbox that I will unsubscribe from MailMate. So my
> question, is this the only solution?
FWIW, your usage is MUCH smaller than mine. If I had only your situation, I
would not start doing what you are doing... :-) :-) :-)
That said, your plan is exactly what I have implemented.
What I have is a virtual mailbox ("To Archive") which include all email that is
older than one year, with sub mailboxes which is "one month per sub mailbox".
Then I move all mail which is in these "To Archive" sub mailboxes to mailboxes
on this separate account which only have this older mail.
Two important things:
- The actual move, the archiving, is something I do manually
- I do *NOT* move mail that I have flags on, those stay
Implement as follows:
1. Create a virtual mailbox "The Mail" that includes all mail that is to be
archived, for example the following mailboxes. No rules for this mailbox. This
is btw the one you base all actions on (all other mailboxes) instead of "All
Mail". For example "Unread", "Default mailbox for searches" etc.
1.1. Mail NOT in the archive itself (in that account)
1.2. Mail NOT in Draft
1.3. Mail NOT in Junk
1.4. Mail NOT in Deleted Messages
2. Create the mailbox "To Archive" with the following characteristics (2.2.3 is
to not have data for the Apple Notes application be moved away):
2.1. Act on "The Mail" as the mailbox
2.2. Conditions, where all of the following are true:
2.2.1. "Date is not within 1 year"
2.2.2. "Message is not flagged"
2.2.3. "X-Uniform-Type-Identifier" is not com.apple.mail-note
2.3. For sub mailboxes set:
2.3.1. Have a sub mailbox for each unique value of Date->Month
2.3.2. Set name of sub mailbox to "${#date.month}"
Then you create one folder in your archive account which have the same name as
the sub mailboxes within "To Archive", open the sub mailbox, select all, and
just drag the mail over.
This 2nd account, with the archive you then do not have to for example add on
your phone etc. And more importantly, as you act all Mailmate commands and
virtual folders on the "The Mail" folder, the speed up with be "a lot".
Patrik
P.S. FWIW, I get 12-13k messages a month, my MailMate process is 5-6G and I
have a total of 2.8 million email messages by now...adding 12-15k per month.
signature.asc
Description: OpenPGP digital signature
___
mailmate mailing list
mailmate@lists.freron.com
https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
On 9 May 2020, at 16:32, Bill Cole wrote: On 9 May 2020, at 15:45, Travis Risner wrote: Hi Guillaume, I have no direct knowledge of how MM works, but I have suspicions. If MM is __synchronizing__ with all the IMAP servers, then I suspect that it has to do something for each of the 200,000 (or 500,000) emails to verify that it is still there and hasn’t changed. One fundamental rule of IMAP is that messages never change. Message flags can change, but the combination of mailbox name, UIDVALIDITY value for the mailbox, and UID must refer to a single immutable message on that server forever. Re-downloading a message only happens if the client or server has a major failure that forces a rebuild of its message indices. Also (and please correct me if I'm wrong), if you store messages in mailboxes other than INBOX (like, an Archive or some such), and you've not subscribed to that mailbox, MM will never even look at it, during its normal duties, until you move something in there, right? ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
On 9 May 2020, at 15:45, Travis Risner wrote: Hi Guillaume, I have no direct knowledge of how MM works, but I have suspicions. If MM is __synchronizing__ with all the IMAP servers, then I suspect that it has to do something for each of the 200,000 (or 500,000) emails to verify that it is still there and hasn’t changed. One fundamental rule of IMAP is that messages never change. Message flags can change, but the combination of mailbox name, UIDVALIDITY value for the mailbox, and UID must refer to a single immutable message on that server forever. Re-downloading a message only happens if the client or server has a major failure that forces a rebuild of its message indices. Maybe it downloads each email and compares checksums. Maybe it has a clever way of interacting with the IMAP server so it does not have to download every byte of each email. Even so, MM still has to do something for each email every time it checks. That might be what is chewing up all the bandwidth. IMAP synchronization does not usually require examination of every message. Instead, when a SELECT command is executed for a mailbox, the server responds with mailbox metadata which is roughly 400 bytes total, describing the state of the mailbox and its messages. This allows the client to determine what it needs to do to synchronize the mailbox. If nothing in a mailbox has changed since last sync and the server supports the "CONDSTORE" extension (as most do,) the client doesn't even need to look at any message flags. Without CONDSTORE, reliable synchronization requires the client to check the flags of all messages in a mailbox, which is a few dozen bytes per message in extreme cases. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not For Hire (currently) ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Hi Guillaume, I have no direct knowledge of how MM works, but I have suspicions. If MM is __synchronizing__ with all the IMAP servers, then I suspect that it has to do something for each of the 200,000 (or 500,000) emails to verify that it is still there and hasn’t changed. Maybe it downloads each email and compares checksums. Maybe it has a clever way of interacting with the IMAP server so it does not have to download every byte of each email. Even so, MM still has to do something for each email every time it checks. That might be what is chewing up all the bandwidth. My solution is similar to Tracy’s. I run MM on my laptop and run a different email program (Thunderbird) on my “server” at home. That email program has rules that will transfer selected emails (that I don’t need to see immediately) to local email folders and remove those emails from the IMAP server. For other emails that I have looked at but don’t need online any more, I manually transfer to other local email folders which also removes them from the IMAP server. Thus, the only emails I keep on the IMAP servers are the ones that I want to do something about later. I too will be looking at Mail Steward and Horcrux to see if that is a better solution. HTH, Travis -- Travis Risner On 5/9/20 2:46 PM, Guillaume Barrette wrote: Dear Tracy, Thanks for your reply, this is greatly appreciated! Yes, I understand around 200,000 emails may be quite a lot and no I don't need all of them instantly and yes I was thinking of doing something to reduce that (backup + moving emails to submailboxes that would be unsubscribed). However, I saw some other mentions of people having 200,000 or 250,000 or even 500,000 emails in MailMate, so I was wondering if my count was that high before making the move. On my side, I don't find MailMate really slow with that amount of emails, it's really regarding the bandwidth + disk space + RAM. I know the disk space + RAM would shrink by having fewer emails, but was wondering if the bandwidth would be the same since the only emails that are touched are the recent ones, so does removing the old ones will really reduce the bandwidth or it is simply how it works and if so I'll need to find a way around (maybe raising the delay for the Synchronization Schedule of most mailboxes will help or creating a script to toggle the Online/Offline state of the mailboxes could help...) With other email clients I wasn't synchronizing all those years for all my email accounts since there was a feature to only synchronize X months, but I like MailMate and I'm sure I'll find a way to put things in a shape that I like, but I was wondering regarding the different options that I could approach those points before making a big move. With that said, thanks for giving an insight on your workflow and mentioning MailSteward and Horcrux (I didn't know this last one), I may go for one of them. Thanks for your help, -- Guillaume On 9 May 2020, at 13:31, Tracy Valleau wrote: Hello, Do you really need 200,000 emails to be instantly available to you, or are you using 99% of that just to store old emails? If the latter, then you could inprove your situation immensely by using an email archiver, such as Horcrux, or (my preferred) MailSteward (which I have been using for well over a decade.) These will move your emails into a database, and allow you to remove them from your server (ie: delete them). I filter out my spam first, and then use MailSteward to archive the rest. It has never failed me, and is quite fast at finding emails. My collection goes back to 1993. You can use either the SQLite version, or the MySQL version. I used MySQL for a very long time, and then realized that I really never referred to 20-year old emails, so I put them into a SQLite table, and saved it out separately. I can load it in if I ever need to find an old email, and now I keep only the past 6 or so years active in MailSteward at any one time. My own copy of MailMail usually runs a total < 50 active emails, so it is lightning fast, and resource light. If I really need to see an old email, I just run MailSteward and look it up. In practice, I may do that twice a week or so. Note that this works because while I have probably 30 email addresses on my server (I'm a developer), I have them all forward to a single mailbox. I did that originally because that way I only needed one email account in MailMate (the account everything is forwarded to) and MM can still filter everything based on the original address a given email was sent to. MUCH more efficient than checking 30 different accounts from my computer! AND, that in turn allows me to use MailSteward, which is designed for Apple Mail, but conveniently has a "also collect emails from this folder" which I have filled with my local storage: "file:///Users/tracyv/Library/Application%20Support/MailMate/Messages/IMAP/tracy%2540mymail...
Re: [MlMt] Ideas to reduce the footprint of MailMate
2 GB of network traffic per day seems high. I’m going to leave Activity monitor open on my MailMate today and see what I get. Messages.noindex tends to be a bit larger than the on-server size of the messages, but not staggeringly so. For example, one of my accounts is using 4.7 GB on the server but its folder inside Messages.noindex is 5.0 GB. Also I’d suggest paying attention to the “Private memory” rather than “Memory” as that shows the amount of RAM MailMate is using that isn’t shared with other processes. (If I understand correctly.) Hope this helps. -sam On 9 May 2020, at 10:45, Guillaume Barrette wrote: Dear MailMate Users, First, I want to say that I really like MailMate with its customization and search possibilities, this is not a rant, but just that I would like to tune it to reduce its footprint if possible. In short I find it a little heavy regarding its usage of resources. To give some statistics: My mailboxes consist of: - 195,766 messages collected from 5 mailboxes (3 from Gmail + 2 other from my web host which have pretty much no message in them) - I'm getting around 80 emails per day What I would like to improve: 1. By investigating MailMate's downloads from the Activity Monitor Network tab, I see that MailMate is downloading around 2gb of data per days 2. My "Messages.noindex" directory takes around 20gb of space on my drive 3. MailMate takes around 1gb of memory while running So, I think my best way to reduce most of it would be to move a lot of my emails to a new submailbox that I will unsubscribe from MailMate. So my question, is this the only solution? I know there's no way to set MailMate to only download emails from the "last 6 months" or something like that, but is there other solutions than to move everything out? Regarding my MailMate download statistic, my Internet plan consists of 150gb per month, so almost half of it goes to MailMate (2gb x 30 days = 60gb / 150gb), is this normal our could it be a bug? I don't think my previous email application was using that much bandwidth... I have difficulty to see how MailMate can use that much bandwidth since by looking at a big day (105 mails) it's around 6.8 mb of storage space. Is MailMate batch re-downloading many emails every day ? To mention, all my accounts have their "Synchronization Schedule" set to "Every 10 minutes". I have multiple Smart Mailboxes, with some submailboxes populated using the "Submailbox for each unique value of", could this add to the bandwidth and does reducing the amount will greatly help in reducing the memory used or the memory is more proportional to the number of emails? Thanks anyone for any cues and help! All the best, -- Guillaume ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Dear Tracy, Thanks for your reply, this is greatly appreciated! Yes, I understand around 200,000 emails may be quite a lot and no I don't need all of them instantly and yes I was thinking of doing something to reduce that (backup + moving emails to submailboxes that would be unsubscribed). However, I saw some other mentions of people having 200,000 or 250,000 or even 500,000 emails in MailMate, so I was wondering if my count was that high before making the move. On my side, I don't find MailMate really slow with that amount of emails, it's really regarding the bandwidth + disk space + RAM. I know the disk space + RAM would shrink by having fewer emails, but was wondering if the bandwidth would be the same since the only emails that are touched are the recent ones, so does removing the old ones will really reduce the bandwidth or it is simply how it works and if so I'll need to find a way around (maybe raising the delay for the Synchronization Schedule of most mailboxes will help or creating a script to toggle the Online/Offline state of the mailboxes could help...) With other email clients I wasn't synchronizing all those years for all my email accounts since there was a feature to only synchronize X months, but I like MailMate and I'm sure I'll find a way to put things in a shape that I like, but I was wondering regarding the different options that I could approach those points before making a big move. With that said, thanks for giving an insight on your workflow and mentioning MailSteward and Horcrux (I didn't know this last one), I may go for one of them. Thanks for your help, -- Guillaume On 9 May 2020, at 13:31, Tracy Valleau wrote: Hello, Do you really need 200,000 emails to be instantly available to you, or are you using 99% of that just to store old emails? If the latter, then you could inprove your situation immensely by using an email archiver, such as Horcrux, or (my preferred) MailSteward (which I have been using for well over a decade.) These will move your emails into a database, and allow you to remove them from your server (ie: delete them). I filter out my spam first, and then use MailSteward to archive the rest. It has never failed me, and is quite fast at finding emails. My collection goes back to 1993. You can use either the SQLite version, or the MySQL version. I used MySQL for a very long time, and then realized that I really never referred to 20-year old emails, so I put them into a SQLite table, and saved it out separately. I can load it in if I ever need to find an old email, and now I keep only the past 6 or so years active in MailSteward at any one time. My own copy of MailMail usually runs a total < 50 active emails, so it is lightning fast, and resource light. If I really need to see an old email, I just run MailSteward and look it up. In practice, I may do that twice a week or so. Note that this works because while I have probably 30 email addresses on my server (I'm a developer), I have them all forward to a single mailbox. I did that originally because that way I only needed one email account in MailMate (the account everything is forwarded to) and MM can still filter everything based on the original address a given email was sent to. MUCH more efficient than checking 30 different accounts from my computer! AND, that in turn allows me to use MailSteward, which is designed for Apple Mail, but conveniently has a "also collect emails from this folder" which I have filled with my local storage: "file:///Users/tracyv/Library/Application%20Support/MailMate/Messages/IMAP/tracy%2540mymail@mail.mymail.org/INBOX.mailbox/dreamhost.mailbox/" Upshot? I have every non-spam email I've received for the past 27 years, and I can find any given one of them almost instantly. Works for me, but of course YMMV. HTH www.valleau.art ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
[MlMt] Ideas to reduce the footprint of MailMate
Hello, Do you really need 200,000 emails to be instantly available to you, or are you using 99% of that just to store old emails? If the latter, then you could inprove your situation immensely by using an email archiver, such as Horcrux, or (my preferred) MailSteward (which I have been using for well over a decade.) These will move your emails into a database, and allow you to remove them from your server (ie: delete them). I filter out my spam first, and then use MailSteward to archive the rest. It has never failed me, and is quite fast at finding emails. My collection goes back to 1993. You can use either the SQLite version, or the MySQL version. I used MySQL for a very long time, and then realized that I really never referred to 20-year old emails, so I put them into a SQLite table, and saved it out separately. I can load it in if I ever need to find an old email, and now I keep only the past 6 or so years active in MailSteward at any one time. My own copy of MailMail usually runs a total < 50 active emails, so it is lightning fast, and resource light. If I really need to see an old email, I just run MailSteward and look it up. In practice, I may do that twice a week or so. Note that this works because while I have probably 30 email addresses on my server (I'm a developer), I have them all forward to a single mailbox. I did that originally because that way I only needed one email account in MailMate (the account everything is forwarded to) and MM can still filter everything based on the original address a given email was sent to. MUCH more efficient than checking 30 different accounts from my computer! AND, that in turn allows me to use MailSteward, which is designed for Apple Mail, but conveniently has a "also collect emails from this folder" which I have filled with my local storage: "file:///Users/tracyv/Library/Application%20Support/MailMate/Messages/IMAP/tracy%2540mymail@mail.mymail.org/INBOX.mailbox/dreamhost.mailbox/" Upshot? I have every non-spam email I've received for the past 27 years, and I can find any given one of them almost instantly. Works for me, but of course YMMV. HTH www.valleau.art ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Sorry, I forgot to specify... I'm using the last non-beta version of MailMate: Version 1.13.1 (5671) -- Guillaume On 9 May 2020, at 10:57, Guillaume Barrette wrote: Hey Alexandre, Let me know what you found out on your side! It would be great to see if it's my setup the problem. To give you some tips, open the Activity Monitor and go into the Network Tab. Keep the application open on this state or the bandwidth will reset to 0. You can filter to MailMate Or you can use the Terminal.app and run: nettop There you can scroll to see MailMate and see its bytes_in for the download. Best, -- Guillaume On 9 May 2020, at 10:49, Alexandre Takacs wrote: On 9 May 2020, at 16:45, Guillaume Barrette wrote: Thanks anyone for any cues and help! I'm afraid I don't have anything actionable but I will have a closer look to MM and bandwidth usage. I have always felt (and complained) that MM was a resource hog but never actually took the time to get into taking measurements. Let's do this... A. Takacs Augicom SA +41 (22) 301 16 00 ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
Hey Alexandre, Let me know what you found out on your side! It would be great to see if it's my setup the problem. To give you some tips, open the Activity Monitor and go into the Network Tab. Keep the application open on this state or the bandwidth will reset to 0. You can filter to MailMate Or you can use the Terminal.app and run: nettop There you can scroll to see MailMate and see its bytes_in for the download. Best, -- Guillaume On 9 May 2020, at 10:49, Alexandre Takacs wrote: On 9 May 2020, at 16:45, Guillaume Barrette wrote: Thanks anyone for any cues and help! I'm afraid I don't have anything actionable but I will have a closer look to MM and bandwidth usage. I have always felt (and complained) that MM was a resource hog but never actually took the time to get into taking measurements. Let's do this... A. Takacs Augicom SA +41 (22) 301 16 00 ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
Re: [MlMt] Ideas to reduce the footprint of MailMate
On 9 May 2020, at 16:45, Guillaume Barrette wrote: Thanks anyone for any cues and help! I'm afraid I don't have anything actionable but I will have a closer look to MM and bandwidth usage. I have always felt (and complained) that MM was a resource hog but never actually took the time to get into taking measurements. Let's do this... A. Takacs Augicom SA +41 (22) 301 16 00 ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
[MlMt] Ideas to reduce the footprint of MailMate
Dear MailMate Users, First, I want to say that I really like MailMate with its customization and search possibilities, this is not a rant, but just that I would like to tune it to reduce its footprint if possible. In short I find it a little heavy regarding its usage of resources. To give some statistics: My mailboxes consist of: - 195,766 messages collected from 5 mailboxes (3 from Gmail + 2 other from my web host which have pretty much no message in them) - I'm getting around 80 emails per day What I would like to improve: 1. By investigating MailMate's downloads from the Activity Monitor Network tab, I see that MailMate is downloading around 2gb of data per days 2. My "Messages.noindex" directory takes around 20gb of space on my drive 3. MailMate takes around 1gb of memory while running So, I think my best way to reduce most of it would be to move a lot of my emails to a new submailbox that I will unsubscribe from MailMate. So my question, is this the only solution? I know there's no way to set MailMate to only download emails from the "last 6 months" or something like that, but is there other solutions than to move everything out? Regarding my MailMate download statistic, my Internet plan consists of 150gb per month, so almost half of it goes to MailMate (2gb x 30 days = 60gb / 150gb), is this normal our could it be a bug? I don't think my previous email application was using that much bandwidth... I have difficulty to see how MailMate can use that much bandwidth since by looking at a big day (105 mails) it's around 6.8 mb of storage space. Is MailMate batch re-downloading many emails every day ? To mention, all my accounts have their "Synchronization Schedule" set to "Every 10 minutes". I have multiple Smart Mailboxes, with some submailboxes populated using the "Submailbox for each unique value of", could this add to the bandwidth and does reducing the amount will greatly help in reducing the memory used or the memory is more proportional to the number of emails? Thanks anyone for any cues and help! All the best, -- Guillaume ___ mailmate mailing list mailmate@lists.freron.com https://lists.freron.com/listinfo/mailmate
