Re: [mailop] IMAP to IMAP

[Brandon Long via mailop]

GSuite provides a number of different migration tools, you don't have to
roll your own.

https://support.google.com/a/answer/6251069?hl=en

If it's to consumer accounts, you may need to roll your own.  You can use
the pop fetcher, but that won't maintain the folder structure (and assumes
you can present the full mailbox to pop, that's the default with Gmail, but
may not be the case for other IMAP services).  I kept meaning to have a
header which could specify the labels to apply, but never got around to it.

Also, for GSuite, you can avoid collecting the passwords of the destination
accounts and use OAUTH (via SASL OAUTHBEARER) with a service account.
Unfortunately, regular IMAP AUTH on Gmail doesn't support separate
authz/authn with passwords.

https://developers.google.com/identity/protocols/OAuth2ServiceAccount

hmm, doesn't look like they wrote an OAUTHBEARER page for Gmail IMAP,
there's just the older XOAUTH2 one:
https://developers.google.com/gmail/imap/xoauth2-protocol

XOAUTH2 was from a draft release of OAUTHBEARER, so the changes should be
minimal.  That page also talks about domain-wide delegation.

The limits you're referring to are listed here:
https://support.google.com/a/answer/1071518?hl=en

The real limits are more complicated than that, the ones listed there are
the daily limits. There are also longer term limits which are related to
the size of the mailbox.
Ie, for download, once you exceed the daily limit, you start working on the
longer limit, which allows for downloading the entire mailbox about 5 times
in 3 months (ie, it's related to the mailbox size, and the three months is
the time it takes for the full limit to regen, but it's not a linear
thing).  The complicated nature of those limits are one reason they're not
listed on that page, though it mentions you can download the mailbox 3
times (the code is I think 5.5, but with All Mail, some clients will
download everything twice...)

The same is true for the uploads, which is also related to the existing
size of the mailbox, which is itself kind of weird (the more you upload,
the more you can upload...)

If you really want, you can also go the route of writing your own using the
Gmail API, not sure if that gains you much, though.


Brandon


On Fri, Dec 15, 2017 at 2:53 PM John Levine  wrote:

> In article  you write:
> >On 12/15/2017 12:27 PM, Steve Atkins wrote:
> >imapsync has a --gmail flag that handles all of the uniquenesses of
> >Gmail. I've used it many times to migrate things.
>
> The target is indeed gmail.  The client has 20 users, most of whom
> have between 5 and 15 GB in their various subfolders.  (It's a design
> firm, they send around a lot of autocad files.)
>
> Gmail says they have a 500 MB/day IMAP upload limit, and imapsync will
> throttle to stay under that limit so a 10 GB mailbox would take three
> weeks.  Really?  Ugh.
>
> R's,
> JOhn
> --
> Regards,
> John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[John Levine]

In article  you write:
>On 12/15/2017 12:27 PM, Steve Atkins wrote:
>imapsync has a --gmail flag that handles all of the uniquenesses of 
>Gmail. I've used it many times to migrate things.

The target is indeed gmail.  The client has 20 users, most of whom
have between 5 and 15 GB in their various subfolders.  (It's a design
firm, they send around a lot of autocad files.)

Gmail says they have a 500 MB/day IMAP upload limit, and imapsync will
throttle to stay under that limit so a 10 GB mailbox would take three
weeks.  Really?  Ugh.

R's,
JOhn
-- 
Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Michael Peddemors]

On 17-12-15 12:07 PM, John Levine wrote:

I have a client who's moving from one mail system to another, and has
quite a lot of mail on the old system's IMAP server that they want to
take with them.

While I can certainly write a python script that enumerates the
mailboxes and copies stuff, I was hoping someone else already had.

R's,
John



Google 'imapsync', it is the goto tool for that..


--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic

A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Kurt Andersen (b)]

On Fri, Dec 15, 2017 at 8:18 PM, Glendon Solsberry  wrote:

> https://github.com/imapsync/imapsync
>
> I do believe it requires knowledge of the users passwords on both systems.
>
> Note specifically the "HUGE MIGRATION" section.
>

I can vouch for imapsync. It was used to migrate a very huge number of
mailboxes. The biggest caveat has to do with non-standard/deficient IMAP
implementations that you might have to wrestle with. For a single user, it
should be pretty straightforward.

--Kurt
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Al Iverson]

On Fri, Dec 15, 2017 at 3:54 PM, Ted Cabeen  wrote:
> On 12/15/2017 12:27 PM, Steve Atkins wrote:
>>
>> https://github.com/imapsync/imapsync or _maybe_
>> https://github.com/wrzlbrmft/imapcopy
>>
>> If the destination system is dovecot then they should probably use dsync.
>>
>> If the destination system is a Large Webmail Provider then they should
>> probably use the providers "import a pop3 account" tool if possible.
>
>
> That will only move INBOX, not sub-folders.
>
> imapsync has a --gmail flag that handles all of the uniquenesses of Gmail.
> I've used it many times to migrate things.

I actually used Google's POP3 polling to transfer about 364,000
messages from one Gmail account to another recently. It didn't only
move the inbox.

The only downsides were, it is very slow -- this took more than a
week. I didn't care, I had time.
And also, some messages that have suspicious content or attachments,
they do not get transferred over. Instead, Gmail replaces the body of
that message, in the new account, with a warning. Since I work in
email abuse / anti-spam, I was potentially more affected by that
versus your average Joe. Still, it wasn't too many messages, and I
decided to just let them go.

Cheers,
Al


-- 
al iverson // wombatmail // miami
http://www.aliverson.com
http://www.spamresource.com

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Ted Cabeen]

On 12/15/2017 12:27 PM, Steve Atkins wrote:

https://github.com/imapsync/imapsync or _maybe_ 
https://github.com/wrzlbrmft/imapcopy

If the destination system is dovecot then they should probably use dsync.

If the destination system is a Large Webmail Provider then they should probably use the 
providers "import a pop3 account" tool if possible.


That will only move INBOX, not sub-folders.

imapsync has a --gmail flag that handles all of the uniquenesses of 
Gmail. I've used it many times to migrate things.


--Ted


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Steve Atkins]

> On Dec 15, 2017, at 12:07 PM, John Levine  wrote:
> 
> I have a client who's moving from one mail system to another, and has
> quite a lot of mail on the old system's IMAP server that they want to
> take with them.
> 
> While I can certainly write a python script that enumerates the
> mailboxes and copies stuff, I was hoping someone else already had.


https://github.com/imapsync/imapsync or _maybe_ 
https://github.com/wrzlbrmft/imapcopy

If the destination system is dovecot then they should probably use dsync.

If the destination system is a Large Webmail Provider then they should probably 
use the providers "import a pop3 account" tool if possible.

Cheers,
  Steve


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Aaron Richton]

On Fri, 15 Dec 2017, John Levine wrote:

I have a client who's moving from one mail system to another, and has 
quite a lot of mail on the old system's IMAP server that they want to 
take with them.


While I can certainly write a python script that enumerates the 
mailboxes and copies stuff, I was hoping someone else already had.


Take a look at https://wiki2.dovecot.org/Tools/Doveadm/Sync -- pretty sure 
you can set up imapc to imapc, although I've never done it personally.


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Glendon Solsberry]

https://github.com/imapsync/imapsync 

I do believe it requires knowledge of the users passwords on both systems.

Note specifically the "HUGE MIGRATION" section.


> On Dec 15, 2017, at 3:07 PM, John Levine  wrote:
> 
> I have a client who's moving from one mail system to another, and has
> quite a lot of mail on the old system's IMAP server that they want to
> take with them.
> 
> While I can certainly write a python script that enumerates the
> mailboxes and copies stuff, I was hoping someone else already had.
> 
> R's,
> John
> -- 
> Regards,
> John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for 
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly
> 
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

[mailop] IMAP to IMAP

[John Levine]

I have a client who's moving from one mail system to another, and has
quite a lot of mail on the old system's IMAP server that they want to
take with them.

While I can certainly write a python script that enumerates the
mailboxes and copies stuff, I was hoping someone else already had.

R's,
John
-- 
Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] SPF recommendations

[Michael Peddemors]

And for my feedback..

We use -all for important domains, involved in ecommerce or confidential 
data.  And yes, sometimes we get a bounce, because someone forwarded 
their email to another party, but it is rare.. (and forwarding should be 
discouraged).


However, it is better than the risk of abuse.  Not that a great fan or 
proponent of SPF everywhere, or think it is the end all.


And we also do 'some' recognition of failed SPF when we do filtering, 
but not so much.  However, when it comes to big banks and big companies 
where 'phishing' is common, we do use it to seed our Known Sender 
Forgery system, but in those cases we actually treat -all and ~all quite 
similar.


Companies of this size SHOULD know the dangers of phishing, and SHOULD 
have accurate SPF records.. And yes, it COULD trigger a rejection notice 
when coming through a forwarder, but should you forward something as 
sensitive as banking information ;)


But it is only a 'small' part of the overall arsenal.  Doesn't help much 
when 'phish' attacks use a domain with a small typo difference.. eg all 
the fake paypal/apple domains getting registered on a regular basis.





On 17-12-15 06:12 AM, Al Iverson wrote:

You're not wrong. I would only say say that perhaps this makes -all
harmless versus something one truly needs to worry about or avoid.

There's a lot of past, quite possibly bogus, guidance where we were
all pushed as ESP senders to implement -all, given the impression that
once upon a time it provided an indirect deliverability boost in some
places. Inertia is strong.

I still personally want -all for myself, because I think there are
possibly a lot of third or fourth tier smaller ISPs, and hobbyists,
and non-US ISPs, that perhaps have SPF support but aren't there with
DMARC yet.

Cheers,
Al Iverson

On Thu, Dec 14, 2017 at 5:28 PM, Brandon Long  wrote:

My point is that -all is policy, and most people ignore the policy portions
of SPF because it completely fails a lot of forwarding cases.

-all is asking receivers to reject mail that doesn't pass.

~all isn't policy.

In practice, very few receivers implement SPF policy (except -all by itself
for domains which don't send mail as a special case).

Maybe there are some smaller receivers who will pay attention to it, but
you're almost certainly going to get more false positives from them than
real positives.  And you won't even notice.

If you want policy, use DMARC, it's what it's there for, and these things
are considered.  As much as DMARC rightly gets pushback for the parts of
forwarding it fails at, it's definitely more useful for policy goals, and
has much wider adoption.

DKIM, for example, explicitly says that a DKIM fail means nothing.  Which
doesn't prevent folks from rejecting messages with broken DKIM signatures,
probably the same folks who follow
-all.

Brandon


On Thu, Dec 14, 2017 at 12:17 PM Al Iverson  wrote:


On Thu, Dec 14, 2017 at 2:14 PM, Brandon Long via mailop
 wrote:


On Thu, Dec 14, 2017 at 11:09 AM Jim Popovitch  wrote:


On Thu, Dec 14, 2017 at 11:33 AM, Vladimir Dubrovin via mailop
 wrote:


In fact, you should not use "-all" for your mail domain if you care
about deliverability.


FALSE!  (Also, you should not randomly add CC recipients to the same
mailinglist that you are responding to)

Aside from a few HUGE providers, those with very large and disparate
networks/offices/topology

-all means that the domain operator knows what they are doing, knows
what their network consists of and how email is routed within their
network.  It further states that the -all publisher has committed to
staying abreast of what happens in their environment in order to
assure their IP space is properly routing email.  It instills
confidence.

~all is just plain lazy, and is akin to saying that you don't have
confidence in your ability to own and control your own network; and
you want others to spend some level of time/money (in the form of CPU
cycles) analyzing email emitted from your network to determine it's
suitability for deliverability.


Or, it acknowledges the fact that the people you send mail to may
forward
that
mail, and trying to control that is silly.


Yeah, but a fail doesn't magically turn into a pass if you turn -all into
~all.

I don't think either is a universal use case, but I see good reasons
for both ways and it depends on what type of company and mail sender
you are. For me, I think -all makes a lot of sense for marketing
senders and folks really worried about phishing/spoofing. And I see
lots of -all mail get forwarded just fine, thanks to, for example, the
fine folks at Google who write the return path when forwarding. :)

Old school forwarding is still a pain even if you pull SPF out of the
equation, no?

Cheers,
Al

--
al iverson // wombatmail // miami
http://www.aliverson.com
http://www.spamresource.com

Re: [mailop] SPF recommendations (was: Re: Earthlink trouble with our PTR)

[Al Iverson]

You're not wrong. I would only say say that perhaps this makes -all
harmless versus something one truly needs to worry about or avoid.

There's a lot of past, quite possibly bogus, guidance where we were
all pushed as ESP senders to implement -all, given the impression that
once upon a time it provided an indirect deliverability boost in some
places. Inertia is strong.

I still personally want -all for myself, because I think there are
possibly a lot of third or fourth tier smaller ISPs, and hobbyists,
and non-US ISPs, that perhaps have SPF support but aren't there with
DMARC yet.

Cheers,
Al Iverson

On Thu, Dec 14, 2017 at 5:28 PM, Brandon Long  wrote:
> My point is that -all is policy, and most people ignore the policy portions
> of SPF because it completely fails a lot of forwarding cases.
>
> -all is asking receivers to reject mail that doesn't pass.
>
> ~all isn't policy.
>
> In practice, very few receivers implement SPF policy (except -all by itself
> for domains which don't send mail as a special case).
>
> Maybe there are some smaller receivers who will pay attention to it, but
> you're almost certainly going to get more false positives from them than
> real positives.  And you won't even notice.
>
> If you want policy, use DMARC, it's what it's there for, and these things
> are considered.  As much as DMARC rightly gets pushback for the parts of
> forwarding it fails at, it's definitely more useful for policy goals, and
> has much wider adoption.
>
> DKIM, for example, explicitly says that a DKIM fail means nothing.  Which
> doesn't prevent folks from rejecting messages with broken DKIM signatures,
> probably the same folks who follow
> -all.
>
> Brandon
>
>
> On Thu, Dec 14, 2017 at 12:17 PM Al Iverson  wrote:
>>
>> On Thu, Dec 14, 2017 at 2:14 PM, Brandon Long via mailop
>>  wrote:
>> >
>> > On Thu, Dec 14, 2017 at 11:09 AM Jim Popovitch  wrote:
>> >>
>> >> On Thu, Dec 14, 2017 at 11:33 AM, Vladimir Dubrovin via mailop
>> >>  wrote:
>> >> >
>> >> > In fact, you should not use "-all" for your mail domain if you care
>> >> > about deliverability.
>> >>
>> >> FALSE!  (Also, you should not randomly add CC recipients to the same
>> >> mailinglist that you are responding to)
>> >>
>> >> Aside from a few HUGE providers, those with very large and disparate
>> >> networks/offices/topology
>> >>
>> >> -all means that the domain operator knows what they are doing, knows
>> >> what their network consists of and how email is routed within their
>> >> network.  It further states that the -all publisher has committed to
>> >> staying abreast of what happens in their environment in order to
>> >> assure their IP space is properly routing email.  It instills
>> >> confidence.
>> >>
>> >> ~all is just plain lazy, and is akin to saying that you don't have
>> >> confidence in your ability to own and control your own network; and
>> >> you want others to spend some level of time/money (in the form of CPU
>> >> cycles) analyzing email emitted from your network to determine it's
>> >> suitability for deliverability.
>> >
>> > Or, it acknowledges the fact that the people you send mail to may
>> > forward
>> > that
>> > mail, and trying to control that is silly.
>>
>> Yeah, but a fail doesn't magically turn into a pass if you turn -all into
>> ~all.
>>
>> I don't think either is a universal use case, but I see good reasons
>> for both ways and it depends on what type of company and mail sender
>> you are. For me, I think -all makes a lot of sense for marketing
>> senders and folks really worried about phishing/spoofing. And I see
>> lots of -all mail get forwarded just fine, thanks to, for example, the
>> fine folks at Google who write the return path when forwarding. :)
>>
>> Old school forwarding is still a pain even if you pull SPF out of the
>> equation, no?
>>
>> Cheers,
>> Al
>>
>> --
>> al iverson // wombatmail // miami
>> http://www.aliverson.com
>> http://www.spamresource.com
>>
>> ___
>> mailop mailing list
>> mailop@mailop.org
>> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop



-- 
al iverson // wombatmail // miami
http://www.aliverson.com
http://www.spamresource.com

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] SPF recommendations

[Philip Paeps]

On 2017-12-15 10:06:44 (+1000), Noel Butler wrote:

On 15/12/2017 09:27, Grant Taylor via mailop wrote:

On 12/14/2017 03:28 PM, Brandon Long via mailop wrote:
My point is that -all is policy, and most people ignore the policy 
portions of SPF because it completely fails a lot of forwarding 
cases.


Every postmaster (or organization behind them) has the prerogative to 
run their mail server(s) the way that they want to.


Agreed, if I publish a -all (which I do and have done for a very very 
long time), I expect receivers doing SPF processing of my domains 
messages, to honor that!  Who the hell are they to assume they know my 
network and its senders better than me.


The pros and cons of SPF -all vs. ~all have been discussed often on this 
mailing list (do people read archives anymore?) and the discussion 
always ends up split between the "receivers with many non-techy users 
who just want their mail" and "senders who insist they know where all 
their mail originates".


If you're a large enough receiver, I think you probably have enough 
other data/signals to treat SPF -all fails simply as another signal in a 
more elaborate scoring system.


If you're a small enough sender, you can probably insist that your users 
use your MSAs.


I publish -all for my personal domain because I know all the users and I 
can whitelist plain forwarders (e.g. freebsd.org).  My -all indicates 
that any message with an envelope @trouble.is that does not come from 
one of my listed servers is so much more likely to be a forgery that I 
don't care about the few exceptions.


Depending on their users, everyone will have different policies.

Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop