Re: Recommended tool for migrating IMAP servers

[Davide Marchi]

[..]
Well, I've read the dsync documentation, but this warning has me a 
little worried:


"Make sure destination is exactly as source, deleting/reverting any 
changes in destination if necessary"


This is when you use the 'backup' option. Dsync then makes 1:1 copy of
the source. If you use 'sync -1' option,
it does not delete mails/folders from destination.

Sami


Ah, ok, for the next sync I will try Dsync!

Many thanks again!


davide

Re: Recommended tool for migrating IMAP servers

[Sami Ketola]

> On 13 Dec 2017, at 18.27, Davide Marchi  wrote:
> 
> Sami Ketola wrote:> We run all our migrations using Dovecot internal dsync. 
> Usually using imapc connector to connect to legacy
>> platform.
>> Wqmi
> 
> Many thanks Wqmi!
> Well, I've read the dsync documentation, but this warning has me a little 
> worried:
> 
> "Make sure destination is exactly as source, deleting/reverting any changes 
> in destination if necessary"

This is when you use the 'backup' option. Dsync then makes 1:1 copy of the 
source. If you use 'sync -1' option, 
it does not delete mails/folders from destination.

> So I followed the Imapsync way, and all works fine.

Sure if you are fine with the imapsync limitations.

Sami

Re: Recommended tool for migrating IMAP servers

[Davide Marchi]

Sami Ketola wrote:> We run all our migrations using Dovecot internal 
dsync. Usually using imapc connector to connect to legacy

platform.

Wqmi


Many thanks Wqmi!
Well, I've read the dsync documentation, but this warning has me a 
little worried:


"Make sure destination is exactly as source, deleting/reverting any 
changes in destination if necessary"


So I followed the Imapsync way, and all works fine.

I seem to have understood thatImapsync works as a traditional client,for 
this reason also much easier to use and with less mistakes risk. And 
anyway a software done very well ;-)



x9p wrote:> I do not believe imapsync has license issues. Its written in 
perl and its
hosted on github. You can pay for support if you want. and disable 
stats

uploaded to their servers, via command line.


Well, clear, many thanks x9p!


Many thanks to all!

Davide



--
cosmogoniA
cosmogoniA
n o p r o v a r e n o f a r e o n o n f a r e n o n c e p r o v a r e

Re: Recommended tool for migrating IMAP servers

[x9p]

On Mon, December 11, 2017 9:32 am, Davide Marchi wrote:

>
> However, it seems that Imapsync has license issues and in fact it's not
> included in the Debian repositories.
> Is it to be used anyway or should be avoid?
>

I do not believe imapsync has license issues. Its written in perl and its
hosted on github. You can pay for support if you want. and disable stats
uploaded to their servers, via command line.

> Many thanks again
>
> Davide
>


cheers.

--
x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE
1524 E7EE

Re: Recommended tool for migrating IMAP servers

[Sami Ketola]

> On 11 Dec 2017, at 12.32, Davide Marchi  wrote:
> 
> Il 2017-12-05 07:16 Sami Ketola ha scritto:
> [..]
>> Trust us. We have run multiple migrations at scale of 10+ million users.
>> Sami
> 
> However, it seems that Imapsync has license issues and in fact it's not 
> included in the Debian repositories.
> Is it to be used anyway or should be avoid?

We run all our migrations using Dovecot internal dsync. Usually using imapc 
connector to connect to legacy
platform.

Wqmi

Re: Recommended tool for migrating IMAP servers

[Davide Marchi]

Il 2017-12-05 07:16 Sami Ketola ha scritto:
[..]


Trust us. We have run multiple migrations at scale of 10+ million 
users.


Sami


However, it seems that Imapsync has license issues and in fact it's not 
included in the Debian repositories.

Is it to be used anyway or should be avoid?

Many thanks again

Davide

Re: Recommended tool for migrating IMAP servers

[Webert de Souza Lima]

On Tue, Dec 5, 2017 at 8:54 AM, Sami Ketola  wrote:

>
> Can't really do DNS change to test with one user as DNS change affects all
> users.


Yes, of course. For test purposes one could justchange the system's host
file and simulate a DNS change, after imapsync is complete
I meant a simple test just to see what happens at the client size, to see
what the user itself could notice.

On Tue, Dec 5, 2017 at 8:54 AM, Sami Ketola  wrote:

>
> What we usually do is put layer of dovecot proxies in front of the legacy
> and destination platform and then have database
> with information if user has been migrated already and forward the
> connection to legacy or dovecot
> platform based on that info. That enables us to migrate user-by-user.


Cool, we do that too :D. We might have learned that from you even. In that
way users can use our Webmail from day 1 when migration starts.


On Tue, Dec 5, 2017 at 8:54 AM, Sami Ketola  wrote:

>
> And both users and mail server admins do notice if there is IMAP UID or
> POP3 UIDL changes. Imagine
> what happens when millions of users suddenly start to redownload terabytes
> of data.
>
> Sami
>

That would just kill my cluster. Happened once due to NFS sharing hang.
Terrible day to be alive.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

Re: Recommended tool for migrating IMAP servers

[María Arrea]

    We migrated 36 TB of mail (65K users) from Sun Messaging to Dovecot 
using Imapsync, worked like a charm.


    Regards

    María

Re: Recommended tool for migrating IMAP servers

[Sebastian Arcus]

On 04/12/17 23:15, Steve Litt wrote:

On Mon, 4 Dec 2017 12:53:15 -0800 (PST)
Joseph Tam  wrote:


"Davide Marchi"  writes:


UW-IMAP's mailutil, imapsync, YippieMove and Larch.


Whatever you use, *don't* use UW-IMAP's mailutil unless you got lots
of time to kill.  It is dreadfully slow -- I used it to export some of
my users' mailboxes to Gmail or other remote mail servers, and I could
almost cut&paste the messages faster.

Like Aki said, if you have the same mailbox format and FS access on
both sides, rsync is much simpler.  You can also try exporting the
old mailboxes via NFS, and with some artful symlinks, march through
your user mailboxes replacing the symlinks with the instantiated
local copies and have almost zero downtime.

Joseph Tam 


Another possibility is to use an email client like Claws-Mail, which is
very fast, create two accounts: One for the old IMAP, one for the new
one, and just copy trees.


I've never used Claws-Mail - so I don't know how well it does. But I've 
done a lot of data migrations between imap servers and accounts - and on 
many occasions I had to use Outlook or Thunderbird to do it. It is 
extremely inefficient, specially if the user has a large number of 
subfolders - as neither software seem to be able to copy subfolders in 
block. The other problem I stumbled on is that you can initiate a large 
transfer (5000 emails at once), the connection, the client or even the 
server chokes at some point for various reasons - and more often than 
not it restarts the transfer from the beginning - and you end up with 
lots of duplications. Specially if transferring through a non-local, 
slower link (ADSL or something similar), I had to end up moving blocks 
of a few hundred emails at a time - then rinse and repeat. Not very 
happy memories when you are dealing with 30,000 emails mailboxes with 
100 subfolders!


To answer the OP, if you have full access to the destination server, and 
the two servers are not Maildir or mbox storage (so copying directly is 
not an option), another possibility might be to use getmail. I am about 
to test this in a few weeks time. Getmail has an option to download 
email from all folders of the originating account. I think I would 
script it with the following options:


1. Download 100 emails at a time - so that if something goes wrong it 
doesn't risk screwing up a large batch (getmail has this option).

2. Download emails from all folders of the account.
3. Delete emails from source after downloading - so that if things go 
wrong, or you have to interrupt the process, you know what has already 
been transferred.

4. Check for success exit code, and repeat until everything is copied.

I haven't tried the above yet, but I think it should be quite robust and 
efficient, and able to run unattended once started.

Re: Recommended tool for migrating IMAP servers

[Sami Ketola]

> On 5 Dec 2017, at 12.46, Webert de Souza Lima  wrote:
> 
> Sorry, I might be wrong about cache invalidation, indeed. What I'm sure is
> that users will hardly notice any server change.
> We've never had user complaints about mailboxes resyncing or anything like
> that after imapsync'ing to a new server, that's why I'd recommend using
> imapsync with no worries.
> 
> It's easy enough to test this on a single user account first and see how is
> imap client's behavior after the DNS change.
> 

Can't really do DNS change to test with one user as DNS change affects all 
users. What we usually
do is put layer of dovecot proxies in front of the legacy and destination 
platform and then have database
with information if user has been migrated already and forward the connection 
to legacy or dovecot
platform based on that info. That enables us to migrate user-by-user.

And both users and mail server admins do notice if there is IMAP UID or POP3 
UIDL changes. Imagine
what happens when millions of users suddenly start to redownload terabytes of 
data.

Sami

Re: Recommended tool for migrating IMAP servers

[Webert de Souza Lima]

On Tue, Dec 5, 2017 at 4:16 AM, Sami Ketola  wrote:

> On Mon, Dec 4, 2017 at 9:16 AM, Sami Ketola 
>  wrote:
>
>>
>> With every other tool you will face end users needing to  invalidate
>> their local caches and
>> redownloading all headers if not also all mail bodies.
>>
>> Sami
>>
>> > On 4 Dec 2017, at 19.59, Webert de Souza Lima 
> wrote:
> >
> > I don't think so. Been using imapsync for large scale migrations from
> > external servers to our dovecot setup. Users don't even see it when the
> key
> > is switched (DNS changes).
> > Go for it.
>
> You are wrong. There is no way to assign IMAP UID:s over IMAP protocol. It
> simply does not support it.
> With imapsync there is absolutely no way to preserve them and you will
> face problems with IMAP UID:s
> not matching the cached mail anymore.
>
> Trust us. We have run multiple migrations at scale of 10+ million users.
>
> Sami


Sorry, I might be wrong about cache invalidation, indeed. What I'm sure is
that users will hardly notice any server change.
We've never had user complaints about mailboxes resyncing or anything like
that after imapsync'ing to a new server, that's why I'd recommend using
imapsync with no worries.

It's easy enough to test this on a single user account first and see how is
imap client's behavior after the DNS change.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

Re: Recommended tool for migrating IMAP servers

[Sami Ketola]

> On 5 Dec 2017, at 10.20, Paolo  wrote:
> 
> Il 04/12/2017 17:37, Michael Slusarz ha scritto:
>>> I think Davide was asking about dsync. If so, the answer is no: dsync
>>> works only when both servers are Dovecot and needs some additional
>>> configuration to work through the network (see
>>> https://wiki2.dovecot.org/Replication).
>> This is entirely incorrect.  The source platform for dsync can be ANY 
>> IMAP/POP server.
>> 
>> The recommended tool for migrating into Dovecot is dsync.  You don't need 
>> any other tool, and other tools aren't going to preserve state so they are 
>> pretty much worthless for a real-world in-place migration.
> Entirely?! At most half incorrect. My apologies, I didn't know about the 
> dsync's IMAP feature (I suppose it wasn't there from the beginning).
> This fact still remain: you can use dsync only when dovecot is involved.
> Even so (IMHO) the sentence "You don't need any other tool" is a bit too 
> much. Who ever know all possible use cases in the world?
> Cheers
> Paolo

IMAPC storage driver has been part of dovecot since January 2011:

https://github.com/dovecot/core/commit/9fb018dea4e2073639249ea8a14ae27cab2c0aac 


Sami

Re: Recommended tool for migrating IMAP servers

[Paolo]

Il 04/12/2017 17:37, Michael Slusarz ha scritto:

I think Davide was asking about dsync. If so, the answer is no: dsync
works only when both servers are Dovecot and needs some additional
configuration to work through the network (see
https://wiki2.dovecot.org/Replication).

This is entirely incorrect.  The source platform for dsync can be ANY IMAP/POP 
server.

The recommended tool for migrating into Dovecot is dsync.  You don't need any 
other tool, and other tools aren't going to preserve state so they are pretty 
much worthless for a real-world in-place migration.
Entirely?! At most half incorrect. My apologies, I didn't know about the 
dsync's IMAP feature (I suppose it wasn't there from the beginning).

This fact still remain: you can use dsync only when dovecot is involved.
Even so (IMHO) the sentence "You don't need any other tool" is a bit too 
much. Who ever know all possible use cases in the world?

Cheers
Paolo

Re: Recommended tool for migrating IMAP servers

[Sami Ketola]

> On 4 Dec 2017, at 19.59, Webert de Souza Lima  wrote:
> 
> On Mon, Dec 4, 2017 at 9:16 AM, Sami Ketola  wrote:
> 
>> 
>> With every other tool you will face end users needing to  invalidate their
>> local caches and
>> redownloading all headers if not also all mail bodies.
>> 
>> Sami
>> 
>> 
> I don't think so. Been using imapsync for large scale migrations from
> external servers to our dovecot setup. Users don't even see it when the key
> is switched (DNS changes).
> Go for it.

You are wrong. There is no way to assign IMAP UID:s over IMAP protocol. It 
simply does not support it.
With imapsync there is absolutely no way to preserve them and you will face 
problems with IMAP UID:s
not matching the cached mail anymore.

Trust us. We have run multiple migrations at scale of 10+ million users.

Sami

Re: Recommended tool for migrating IMAP servers

[Steve Litt]

On Mon, 4 Dec 2017 12:53:15 -0800 (PST)
Joseph Tam  wrote:

> "Davide Marchi"  writes:
> 
> >> UW-IMAP's mailutil, imapsync, YippieMove and Larch.  
> 
> Whatever you use, *don't* use UW-IMAP's mailutil unless you got lots
> of time to kill.  It is dreadfully slow -- I used it to export some of
> my users' mailboxes to Gmail or other remote mail servers, and I could
> almost cut&paste the messages faster.
> 
> Like Aki said, if you have the same mailbox format and FS access on
> both sides, rsync is much simpler.  You can also try exporting the
> old mailboxes via NFS, and with some artful symlinks, march through
> your user mailboxes replacing the symlinks with the instantiated
> local copies and have almost zero downtime.
> 
> Joseph Tam 

Another possibility is to use an email client like Claws-Mail, which is
very fast, create two accounts: One for the old IMAP, one for the new
one, and just copy trees.

SteveT

Steve Litt 
November 2017 featured book: Troubleshooting: Just the Facts
http://www.troubleshooters.com/tjust

Re: Recommended tool for migrating IMAP servers

[Joseph Tam]

"Davide Marchi"  writes:


UW-IMAP's mailutil, imapsync, YippieMove and Larch.


Whatever you use, *don't* use UW-IMAP's mailutil unless you got lots
of time to kill.  It is dreadfully slow -- I used it to export some of
my users' mailboxes to Gmail or other remote mail servers, and I could
almost cut&paste the messages faster.

Like Aki said, if you have the same mailbox format and FS access on both
sides, rsync is much simpler.  You can also try exporting the
old mailboxes via NFS, and with some artful symlinks, march through your user
mailboxes replacing the symlinks with the instantiated local copies and
have almost zero downtime.

Joseph Tam 

Re: Recommended tool for migrating IMAP servers

[Webert de Souza Lima]

On Mon, Dec 4, 2017 at 9:16 AM, Sami Ketola  wrote:

>
> With every other tool you will face end users needing to  invalidate their
> local caches and
> redownloading all headers if not also all mail bodies.
>
> Sami
>
>
I don't think so. Been using imapsync for large scale migrations from
external servers to our dovecot setup. Users don't even see it when the key
is switched (DNS changes).
Go for it.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

Re: Recommended tool for migrating IMAP servers

[Michael Slusarz]

> On December 4, 2017 at 8:46 AM Paolo  wrote:
> 
> 
> Il 04/12/2017 14:33, x9p ha scritto:
> >
> >> Can I use this tool even if I do not know the other remote server
> >> typology?
> >>
> > sure. just need both IMAP ports reachable and valid user/pass for both
> > servers.
> I think Davide was asking about dsync. If so, the answer is no: dsync 
> works only when both servers are Dovecot and needs some additional 
> configuration to work through the network (see 
> https://wiki2.dovecot.org/Replication).

This is entirely incorrect.  The source platform for dsync can be ANY IMAP/POP 
server.

The recommended tool for migrating into Dovecot is dsync.  You don't need any 
other tool, and other tools aren't going to preserve state so they are pretty 
much worthless for a real-world in-place migration.

michael

Re: Recommended tool for migrating IMAP servers

[Paolo]

Il 04/12/2017 14:33, x9p ha scritto:



Can I use this tool even if I do not know the other remote server
typology?


sure. just need both IMAP ports reachable and valid user/pass for both
servers.
I think Davide was asking about dsync. If so, the answer is no: dsync 
works only when both servers are Dovecot and needs some additional 
configuration to work through the network (see 
https://wiki2.dovecot.org/Replication).
I don't know about imapsync but I suppose it is a generic IMAP tool that 
replicates mailboxes using IMAP protocol as a client between two servers.


Cheers
Paolo

Re: Recommended tool for migrating IMAP servers

[x9p]

>> Hi,
>>
>> I vouch for imapsync. Have used it in the past with quite a big amount
>> of
>> emails.
>>
>> cheers.
>>
>> x0p
>
> Ah, thanks  x0!
>

welcome!

>
> Can I use this tool even if I do not know the other remote server
> typology?
>

sure. just need both IMAP ports reachable and valid user/pass for both
servers.

cheers.

x9p

Re: Recommended tool for migrating IMAP servers

[Davide Marchi]

[..]

Hi,

I vouch for imapsync. Have used it in the past with quite a big amount 
of

emails.

cheers.

x0p


Ah, thanks  x0!


Also if you have fs access on both servers, and you are using maildir,
plain rsync works just as well.

Aki


no, I've not fs access on both servers! :-/

If you want to preserve IMAP UID:s and possibly also POP3 UIDL:s then 
dovecot internal

dsync is the only tool that can do it.

With every other tool you will face end users needing to  invalidate 
their local caches and

redownloading all headers if not also all mail bodies.

Sami



Can I use this tool even if I do not know the other remote server 
typology?


Many thanks to all!!

Davide



--
firma

cosmogoniA 
n o p r o v a r e n o f a r e o n o n f a r e n o n c e p r o v a r e

Re: Recommended tool for migrating IMAP servers

[Sami Ketola]

> On 3 Dec 2017, at 23.23, Davide Marchi  wrote:
> 
> Hi Friends,
> I would like to ask you a suggestion:
> I need to migrate a imap server to a new one and then dismiss the old one.
> Reading from relative Dovecot documentation page 
> (https://wiki.dovecot.org/Migration), more tools are shown:
> 
> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
> 
> The each mail servers are Linux based, one of this (mine) is Dovecot.
> Based on your experience which of these tools would be preferable to use?

If you want to preserve IMAP UID:s and possibly also POP3 UIDL:s then dovecot 
internal 
dsync is the only tool that can do it. 

With every other tool you will face end users needing to  invalidate their 
local caches and 
redownloading all headers if not also all mail bodies.

Sami

Re: Recommended tool for migrating IMAP servers

[Aki Tuomi]

Also if you have fs access on both servers, and you are using maildir,
plain rsync works just as well.

Aki


On 04.12.2017 00:17, Harondel J. Sibble wrote:
> Imapsync for sure. Have used it for both IMAP to IMAP and IMAP to Exchange 
> migrations. Works great.
>
>
>> On Dec 3, 2017, at 2:08 PM, x9p  wrote:
>>
>> Hi,
>>
>> I vouch for imapsync. Have used it in the past with quite a big amount of
>> emails.
>>
>> cheers.
>>
>> x0p
>>
>>> Hi Friends,
>>> I would like to ask you a suggestion:
>>> I need to migrate a imap server to a new one and then dismiss the old
>>> one.
>>> Reading from relative Dovecot documentation page
>>> (https://wiki.dovecot.org/Migration), more tools are shown:
>>>
>>> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>>>
>>> The each mail servers are Linux based, one of this (mine) is Dovecot.
>>> Based on your experience which of these tools would be preferable to
>>> use?
>>>
>>>
>>> Thank you very much
>>>
>>> Davide
>>>
>>

Re: Recommended tool for migrating IMAP servers

[Harondel J. Sibble]

Imapsync for sure. Have used it for both IMAP to IMAP and IMAP to Exchange 
migrations. Works great.


> On Dec 3, 2017, at 2:08 PM, x9p  wrote:
> 
> Hi,
> 
> I vouch for imapsync. Have used it in the past with quite a big amount of
> emails.
> 
> cheers.
> 
> x0p
> 
>> Hi Friends,
>> I would like to ask you a suggestion:
>> I need to migrate a imap server to a new one and then dismiss the old
>> one.
>> Reading from relative Dovecot documentation page
>> (https://wiki.dovecot.org/Migration), more tools are shown:
>> 
>> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>> 
>> The each mail servers are Linux based, one of this (mine) is Dovecot.
>> Based on your experience which of these tools would be preferable to
>> use?
>> 
>> 
>> Thank you very much
>> 
>> Davide
>> 
> 
> 

Re: Recommended tool for migrating IMAP servers

[x9p]

Hi,

I vouch for imapsync. Have used it in the past with quite a big amount of
emails.

cheers.

x0p

> Hi Friends,
> I would like to ask you a suggestion:
> I need to migrate a imap server to a new one and then dismiss the old
> one.
> Reading from relative Dovecot documentation page
> (https://wiki.dovecot.org/Migration), more tools are shown:
>
> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>
> The each mail servers are Linux based, one of this (mine) is Dovecot.
> Based on your experience which of these tools would be preferable to
> use?
>
>
> Thank you very much
>
> Davide
>