Re: [mailop] IMAP to IMAP

[Graeme Fowler]

I'm late to this particular party, but for reference in my day job (at 
Loughborough Uni) I recently moved our students from a Google (Apps for Edu | 
GSuite) domain to Office 365.

That was 17000+ mailboxes, INBOX, subfolders and all, using imapsync with a lot 
of 'tuning'.

I ran it in parallel on a single CentOS 7 VM with 24GB RAM and 12 cores, using 
delegated access on the O365 side and 'admin' access via XOAUTH2 on the Google 
side, with a maximum of 130 concurrent jobs. In total it took a shade under 5 
days (around 300Mbps bidirectionally).

We didn't hit any notable limits at all at either end, but I did land the 
migration VM in O365's bad books after mis-cueing the authentication end of 
things during the development phase. They really didn't like getting thousands 
of auth failures per hour!

Graeme
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Brandon Long via mailop]

GSuite provides a number of different migration tools, you don't have to
roll your own.

https://support.google.com/a/answer/6251069?hl=en

If it's to consumer accounts, you may need to roll your own.  You can use
the pop fetcher, but that won't maintain the folder structure (and assumes
you can present the full mailbox to pop, that's the default with Gmail, but
may not be the case for other IMAP services).  I kept meaning to have a
header which could specify the labels to apply, but never got around to it.

Also, for GSuite, you can avoid collecting the passwords of the destination
accounts and use OAUTH (via SASL OAUTHBEARER) with a service account.
Unfortunately, regular IMAP AUTH on Gmail doesn't support separate
authz/authn with passwords.

https://developers.google.com/identity/protocols/OAuth2ServiceAccount

hmm, doesn't look like they wrote an OAUTHBEARER page for Gmail IMAP,
there's just the older XOAUTH2 one:
https://developers.google.com/gmail/imap/xoauth2-protocol

XOAUTH2 was from a draft release of OAUTHBEARER, so the changes should be
minimal.  That page also talks about domain-wide delegation.

The limits you're referring to are listed here:
https://support.google.com/a/answer/1071518?hl=en

The real limits are more complicated than that, the ones listed there are
the daily limits. There are also longer term limits which are related to
the size of the mailbox.
Ie, for download, once you exceed the daily limit, you start working on the
longer limit, which allows for downloading the entire mailbox about 5 times
in 3 months (ie, it's related to the mailbox size, and the three months is
the time it takes for the full limit to regen, but it's not a linear
thing).  The complicated nature of those limits are one reason they're not
listed on that page, though it mentions you can download the mailbox 3
times (the code is I think 5.5, but with All Mail, some clients will
download everything twice...)

The same is true for the uploads, which is also related to the existing
size of the mailbox, which is itself kind of weird (the more you upload,
the more you can upload...)

If you really want, you can also go the route of writing your own using the
Gmail API, not sure if that gains you much, though.


Brandon


On Fri, Dec 15, 2017 at 2:53 PM John Levine  wrote:

> In article  you write:
> >On 12/15/2017 12:27 PM, Steve Atkins wrote:
> >imapsync has a --gmail flag that handles all of the uniquenesses of
> >Gmail. I've used it many times to migrate things.
>
> The target is indeed gmail.  The client has 20 users, most of whom
> have between 5 and 15 GB in their various subfolders.  (It's a design
> firm, they send around a lot of autocad files.)
>
> Gmail says they have a 500 MB/day IMAP upload limit, and imapsync will
> throttle to stay under that limit so a 10 GB mailbox would take three
> weeks.  Really?  Ugh.
>
> R's,
> JOhn
> --
> Regards,
> John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[John Levine]

In article  you write:
>On 12/15/2017 12:27 PM, Steve Atkins wrote:
>imapsync has a --gmail flag that handles all of the uniquenesses of 
>Gmail. I've used it many times to migrate things.

The target is indeed gmail.  The client has 20 users, most of whom
have between 5 and 15 GB in their various subfolders.  (It's a design
firm, they send around a lot of autocad files.)

Gmail says they have a 500 MB/day IMAP upload limit, and imapsync will
throttle to stay under that limit so a 10 GB mailbox would take three
weeks.  Really?  Ugh.

R's,
JOhn
-- 
Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Michael Peddemors]

On 17-12-15 12:07 PM, John Levine wrote:

I have a client who's moving from one mail system to another, and has
quite a lot of mail on the old system's IMAP server that they want to
take with them.

While I can certainly write a python script that enumerates the
mailboxes and copies stuff, I was hoping someone else already had.

R's,
John



Google 'imapsync', it is the goto tool for that..


--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic

A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Kurt Andersen (b)]

On Fri, Dec 15, 2017 at 8:18 PM, Glendon Solsberry  wrote:

> https://github.com/imapsync/imapsync
>
> I do believe it requires knowledge of the users passwords on both systems.
>
> Note specifically the "HUGE MIGRATION" section.
>

I can vouch for imapsync. It was used to migrate a very huge number of
mailboxes. The biggest caveat has to do with non-standard/deficient IMAP
implementations that you might have to wrestle with. For a single user, it
should be pretty straightforward.

--Kurt
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Al Iverson]

On Fri, Dec 15, 2017 at 3:54 PM, Ted Cabeen  wrote:
> On 12/15/2017 12:27 PM, Steve Atkins wrote:
>>
>> https://github.com/imapsync/imapsync or _maybe_
>> https://github.com/wrzlbrmft/imapcopy
>>
>> If the destination system is dovecot then they should probably use dsync.
>>
>> If the destination system is a Large Webmail Provider then they should
>> probably use the providers "import a pop3 account" tool if possible.
>
>
> That will only move INBOX, not sub-folders.
>
> imapsync has a --gmail flag that handles all of the uniquenesses of Gmail.
> I've used it many times to migrate things.

I actually used Google's POP3 polling to transfer about 364,000
messages from one Gmail account to another recently. It didn't only
move the inbox.

The only downsides were, it is very slow -- this took more than a
week. I didn't care, I had time.
And also, some messages that have suspicious content or attachments,
they do not get transferred over. Instead, Gmail replaces the body of
that message, in the new account, with a warning. Since I work in
email abuse / anti-spam, I was potentially more affected by that
versus your average Joe. Still, it wasn't too many messages, and I
decided to just let them go.

Cheers,
Al


-- 
al iverson // wombatmail // miami
http://www.aliverson.com
http://www.spamresource.com

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Ted Cabeen]

On 12/15/2017 12:27 PM, Steve Atkins wrote:

https://github.com/imapsync/imapsync or _maybe_ 
https://github.com/wrzlbrmft/imapcopy

If the destination system is dovecot then they should probably use dsync.

If the destination system is a Large Webmail Provider then they should probably use the 
providers "import a pop3 account" tool if possible.


That will only move INBOX, not sub-folders.

imapsync has a --gmail flag that handles all of the uniquenesses of 
Gmail. I've used it many times to migrate things.


--Ted


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Steve Atkins]

> On Dec 15, 2017, at 12:07 PM, John Levine  wrote:
> 
> I have a client who's moving from one mail system to another, and has
> quite a lot of mail on the old system's IMAP server that they want to
> take with them.
> 
> While I can certainly write a python script that enumerates the
> mailboxes and copies stuff, I was hoping someone else already had.


https://github.com/imapsync/imapsync or _maybe_ 
https://github.com/wrzlbrmft/imapcopy

If the destination system is dovecot then they should probably use dsync.

If the destination system is a Large Webmail Provider then they should probably 
use the providers "import a pop3 account" tool if possible.

Cheers,
  Steve


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Aaron Richton]

On Fri, 15 Dec 2017, John Levine wrote:

I have a client who's moving from one mail system to another, and has 
quite a lot of mail on the old system's IMAP server that they want to 
take with them.


While I can certainly write a python script that enumerates the 
mailboxes and copies stuff, I was hoping someone else already had.


Take a look at https://wiki2.dovecot.org/Tools/Doveadm/Sync -- pretty sure 
you can set up imapc to imapc, although I've never done it personally.


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] IMAP to IMAP

[Glendon Solsberry]

https://github.com/imapsync/imapsync 

I do believe it requires knowledge of the users passwords on both systems.

Note specifically the "HUGE MIGRATION" section.


> On Dec 15, 2017, at 3:07 PM, John Levine  wrote:
> 
> I have a client who's moving from one mail system to another, and has
> quite a lot of mail on the old system's IMAP server that they want to
> take with them.
> 
> While I can certainly write a python script that enumerates the
> mailboxes and copies stuff, I was hoping someone else already had.
> 
> R's,
> John
> -- 
> Regards,
> John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for 
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly
> 
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

[mailop] IMAP to IMAP

[John Levine]

I have a client who's moving from one mail system to another, and has
quite a lot of mail on the old system's IMAP server that they want to
take with them.

While I can certainly write a python script that enumerates the
mailboxes and copies stuff, I was hoping someone else already had.

R's,
John
-- 
Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop