Re: server migration

2024-04-10 Thread Gandalf Corvotempesta via dovecot
in my case, 99% of mailboxes are imap

Il gio 11 apr 2024, 00:08 Michael Peddemors via dovecot 
ha scritto:
 Of course, anyone who is stilling using POP (Leave on Server)
 presents a
 different challenge.. Depending on the client, and how the client
 treated the UID of the message..

 The rest should present no issue..

 On 2024-04-10 14:25, Kirill Miazine via dovecot wrote:
 >
 >
 > • Gandalf Corvotempesta via dovecot [2024-04-10 23:18]:
 >> Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via
 dovecot
 >>  ha scritto:
 >>> UIDVALIDITY change
 >>
 >> In which case uidvalidity would change ?
 >
 > if you do rsync, it doesn't. UIDVALIDITY is stored in dovecot-
 uidlist in
 > maildirs, as described in
 > https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#imap-
 uid-mapping
 > ___
 > dovecot mailing list -- dovecot@dovecot.org
 > To unsubscribe send an email to dovecot-le...@dovecot.org


 --
 "Catch the Magic of Linux..."
 -
 ---
 Michael Peddemors, President/CEO LinuxMagic Inc.
 Visit us at http://www.linuxmagic.com @linuxmagic
 A Wizard IT Company - For More Info http://www.wizard.ca
 "LinuxMagic" a Reg. TradeMark of Wizard Tower TechnoServices Ltd.
 -
 ---
 604-682-0300 Beautiful British Columbia, Canada

 ___
 dovecot mailing list -- dovecot@dovecot.org
 To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Michael Peddemors via dovecot
Of course, anyone who is stilling using POP (Leave on Server) presents a 
different challenge.. Depending on the client, and how the client 
treated the UID of the message..


The rest should present no issue..

On 2024-04-10 14:25, Kirill Miazine via dovecot wrote:



• Gandalf Corvotempesta via dovecot [2024-04-10 23:18]:

Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot
 ha scritto:

UIDVALIDITY change


In which case uidvalidity would change ?


if you do rsync, it doesn't. UIDVALIDITY is stored in dovecot-uidlist in 
maildirs, as described in 
https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#imap-uid-mapping

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org



--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Reg. TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Kirill Miazine via dovecot



• Gandalf Corvotempesta via dovecot [2024-04-10 23:18]:

Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot
 ha scritto:

UIDVALIDITY change


In which case uidvalidity would change ?


if you do rsync, it doesn't. UIDVALIDITY is stored in dovecot-uidlist in 
maildirs, as described in 
https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#imap-uid-mapping

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Gandalf Corvotempesta via dovecot
Il giorno mer 10 apr 2024 alle ore 23:12 Kirill Miazine via dovecot
 ha scritto:
> UIDVALIDITY change

In which case uidvalidity would change ?
Manually changes in config file ?
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Kirill Miazine via dovecot



• Gandalf Corvotempesta via dovecot [2024-04-10 22:59]:

What could trigger a new re-download of message ?


UIDVALIDITY change
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Gandalf Corvotempesta via dovecot
Il giorno mer 10 apr 2024 alle ore 22:32 Marc via dovecot
 ha scritto:
> Why? The whole idea about having a LTS distribution is that you almost never 
> need to do this? It is not like the imap/pop/smtp standards are having yearly 
> innovations. Or is this a service you provide for clients?

In my case the server migration has to be done because the old
datacenter is closing and i have to move all datas to a new server on
a different location.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Gandalf Corvotempesta via dovecot
Il giorno mer 10 apr 2024 alle ore 21:40 Kirill Miazine via dovecot
 ha scritto:
> What you describe is exactly what I have been doing since ... forever
>
> - reduce TTL
> - setup new server
> - rsync
> - stop ALL mail services on old server (also anything which might be
> doing deliveries, this is important), kill client connections, if any
> - rsync again
> - update DNS
> - start mail service on new server
> - verify
> - increase TTL

this is what i've planned, but I have to be 1% sure that clients
wont re-download all messages again.
What could trigger a new re-download of message ?
The new server would be able to handle the mails stored with the old
hostname and at the same time handle
the mails stored with the new hostname (and thus different file name)
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Christopher Wensink via dovecot

Can you expand and explain this:

Why? The whole idea about having a LTS distribution is that you almost never 
need to do this?

Can you provide a link for context?

On 4/10/2024 3:25 PM, Marc via dovecot wrote:



• Gandalf Corvotempesta via dovecot [2024-04-10 21:07]:

Guys, any help?

What you describe is exactly what I have been doing since ... forever


Why? The whole idea about having a LTS distribution is that you almost never 
need to do this? It is not like the imap/pop/smtp standards are having yearly 
innovations. Or is this a service you provide for clients?
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org



--
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwens...@five-star-plastics.com
www.five-star-plastics.com

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


RE: server migration

2024-04-10 Thread Marc via dovecot
> 
> 
> 
> • Gandalf Corvotempesta via dovecot [2024-04-10 21:07]:
> > Guys, any help?
> 
> What you describe is exactly what I have been doing since ... forever
> 

Why? The whole idea about having a LTS distribution is that you almost never 
need to do this? It is not like the imap/pop/smtp standards are having yearly 
innovations. Or is this a service you provide for clients?
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


RE: server migration

2024-04-10 Thread Marc via dovecot
> 
> Guys, any help?

this lacks context. 

> Also, what would happen if the new server has a different hostname ?

So put temporary haproxy infront of it?

> Il giorno dom 10 mar 2024 alle ore 14:28 Gandalf Corvotempesta
>  ha scritto:
> >
> > Hi guys
> > I have to migrate around 10k mailboxes from dovecot 2.13 to (i think)
> > the same version but on a different server.

Why not just mount the data on the other server?

> > I have to reduce as much as possible the inconveniences to the users,
> > at least in this (temporary) phase.

Just prepare two vm configs and stop the old and start the new.

> > What do you suggest to move everything ? Same config, same maildir
> > location and rsync everything ?

You have to have a good reason to change storage formats, it is not really 
related to what instagrammers reeling about. What is your good reason to change?

> > Better ideas ? i've thought to use the exact same config on both
> > servers, then start multiple rsync to sync as much as possible and
> > when ready, drop the old dovecot in the old server, rsync the latest
> > changes, and then move the dns pointment from the old ip to the new
> > one.

I would go for just mount the data images on the other server. If that is not 
possible and you don't seem to know what is on the other server. Do the doveadm 
sync.

I would suggest thinking a bit more about this and what you want in the future. 
Because you don't want to do this to often. There was just a discussion about 
these folder names what you might consider looking at. Als the availability of 
2.4, maybe wait?

> > But what about the MUA downloading emails ? I think this would be
> > safe...or there is a chance that some MUA would re-download everything
> > ? This would be unacceptable.
> >

You will see when you are testing. 



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Kirill Miazine via dovecot



• Gandalf Corvotempesta via dovecot [2024-04-10 21:07]:

Guys, any help?


What you describe is exactly what I have been doing since ... forever

- reduce TTL
- setup new server
- rsync
- stop ALL mail services on old server (also anything which might be 
doing deliveries, this is important), kill client connections, if any

- rsync again
- update DNS
- start mail service on new server
- verify
- increase TTL

You mention multiple rsyncs, I wouldn't bother...


Also, what would happen if the new server has a different hostname ?


You'd get new filenames in Maildir, and this is it.


Il giorno dom 10 mar 2024 alle ore 14:28 Gandalf Corvotempesta
 ha scritto:


Hi guys
I have to migrate around 10k mailboxes from dovecot 2.13 to (i think)
the same version but on a different server.

I have to reduce as much as possible the inconveniences to the users,
at least in this (temporary) phase.

What do you suggest to move everything ? Same config, same maildir
location and rsync everything ?

Better ideas ? i've thought to use the exact same config on both
servers, then start multiple rsync to sync as much as possible and
when ready, drop the old dovecot in the old server, rsync the latest
changes, and then move the dns pointment from the old ip to the new
one.

But what about the MUA downloading emails ? I think this would be
safe...or there is a chance that some MUA would re-download everything
? This would be unacceptable.

thank you

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: server migration

2024-04-10 Thread Gandalf Corvotempesta via dovecot
Guys, any help?
Also, what would happen if the new server has a different hostname ?

Il giorno dom 10 mar 2024 alle ore 14:28 Gandalf Corvotempesta
 ha scritto:
>
> Hi guys
> I have to migrate around 10k mailboxes from dovecot 2.13 to (i think)
> the same version but on a different server.
>
> I have to reduce as much as possible the inconveniences to the users,
> at least in this (temporary) phase.
>
> What do you suggest to move everything ? Same config, same maildir
> location and rsync everything ?
>
> Better ideas ? i've thought to use the exact same config on both
> servers, then start multiple rsync to sync as much as possible and
> when ready, drop the old dovecot in the old server, rsync the latest
> changes, and then move the dns pointment from the old ip to the new
> one.
>
> But what about the MUA downloading emails ? I think this would be
> safe...or there is a chance that some MUA would re-download everything
> ? This would be unacceptable.
>
> thank you
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Server migration, password scheme/hashing, argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, salt?

2023-06-24 Thread Robert Lister



Should work then if you have roundcube 1.6.x and php8.2 which is also 
the

bookworm package version.

Depends on your server spec / number of users if you use argon2 over 
bcrypt.


One approach might be to just migrate all users to BLF-CRYPT anyway, and 
then
set the recommended dovecot member settings and selectively change a few 
users
to ARGON2ID to see the impact.  If you stored both hashes in the 
database, this

would allow you to switch back.

If someone gained  write access to the database somehow, they could 
possibly

replace user's password hash with a new one, thereby allowing them
to gain access to user accounts.

Two-factor authentication of course is the way to go with roundcube,
but by default, there's nothing stopping access using the same 
credentials

via IMAP/Submission without the 2FA, so roundcube 2FA isn't effective by
itself if users also have access to IMAP/Submission.

I improved things a bit by using roundcube plugins:-

mmvi/twofactor_webauthn   -  FIDO2/webauthn 2FA.

And: https://github.com/openSUSE/ap4rc

I modified it a bit to allow using the same username and added some
features to it:  https://github.com/listerr/ap4rc/tree/last-access

Ultimately the goal is to eliminate passwords using OAUTH2 etc but not 
quite

there yet.

R.




On 2023-06-24 14:54, David Mehler wrote:

Hello,

Thanks. The other utility I would be using is the Roundcube webmail
password plugin.  Still trying to figure the best option.

More opinions?
Thanks.
Dave.



--
Robert Lister  - email:  r...@lentil.org  - tel: 020 7043 7996
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Server migration, password scheme/hashing, argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, salt?

2023-06-24 Thread David Mehler
Hello,

Thanks. The other utility I would be using is the Roundcube webmail
password plugin.  Still trying to figure the best option.

More opinions?
Thanks.
Dave.


On 6/24/23, Robert Lister  wrote:
>
> I did a similar upgrade, and now in the process of migrating from
> SHA512-CRYPT
> to BLF-CRYPT with an appropriately set rounds, as I think the default
> rounds
> is a little low.
>
> A good write-up on migrating passwords and calculating the rounds:
> https://kaworu.ch/blog/2016/04/20/strong-crypt-scheme-with-dovecot-postfixadmin-and-roundcube/
>
>
> I would take into consideration the following factors when deciding the
> hashing algo.
>
> 1. Other tools/scripts that need to update or check passwords in the
> database,
> for example:
> - roundcube webmail has a plugin to allow users to change their
> password
>   using a variety of methods.
> - postfixadmin
>
> For a long time, bcrypt wasn't natively supported by either the
> version of php
> or underlying OS libs, so these tools had to rely on calling "doveadm
> pw "
> to generate BLF-CRYPT hashes. And assumed that doveadm was available
> on the same server as it.
>
> The latest versions support bcrypt and newer hashing algos natively.
>
> Some tools might rely on the database (mysql/mariadb) to hash
> passwords, so
> this may also be a consideration.
>
> 2. Server load / libs:
>
> - The Dovecot docs:
> https://doc.dovecot.org/configuration_manual/authentication/password_schemes/
>   has this to say on ARGON2I/ARGON2ID:
>
>   "Argon2 is the winner of password hashing competition held at July
> 2015. The password will
>start with $argon2i$ or $argon2id$. You can use -r to tune
> computational complexity,
>minimum is 3. ARGON2ID is only available if your libsodium is
> recent enough.
>ARGON2 can require quite a hefty amount of virtual memory, so we
> recommend that you set
>service auth { vsz_limit = 2G } at least, or more."
>
> There's a good write up of considering the various algos:
>
> https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html
>
> I considered BLF-CRYPT (for the time being) to be strong enough and a
> good balance between compatibility, strength and server load, given the
> number of users etc.
>
> Rob
>
>
> On 2023-06-23 02:14, David Mehler wrote:
>> Hello,
>>
>> I'm migrating to a new server. It's running Debian 11 currently though
>> that's going 12 this weekend. Currently it uses Openssl v3.0.9, and
>> dovecot 2.3.13 and MySQL (in this case Mariadb) for storing user
>> account information v10.6.14. My question is in regards password
>> storage and scheme/encryption/salts.
>>
>> Currently they are stored in Mariadb password field with a type of
>> varchar and a 255 character length, and are stored as SHA512-CRYPT.
>> I'm wondering if I should keep this as is or when I migrate go to
>> another scheme? I'm thinking argon2i, argon2d, argon2id, sha512,
>> sha512-crypt, tiger2, saltt?
>
>
> --
> Robert Lister  - email:  r...@lentil.org  - tel: 020 7043 7996
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
>
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Server migration, password scheme/hashing, argon2i, argon2d, argon2id, sha512, sha512-crypt, tiger2, salt?

2023-06-24 Thread Robert Lister



I did a similar upgrade, and now in the process of migrating from 
SHA512-CRYPT
to BLF-CRYPT with an appropriately set rounds, as I think the default 
rounds

is a little low.

A good write-up on migrating passwords and calculating the rounds:
https://kaworu.ch/blog/2016/04/20/strong-crypt-scheme-with-dovecot-postfixadmin-and-roundcube/


I would take into consideration the following factors when deciding the 
hashing algo.


1. Other tools/scripts that need to update or check passwords in the 
database,

   for example:
   - roundcube webmail has a plugin to allow users to change their 
password

 using a variety of methods.
   - postfixadmin

   For a long time, bcrypt wasn't natively supported by either the 
version of php
   or underlying OS libs, so these tools had to rely on calling "doveadm 
pw "

   to generate BLF-CRYPT hashes. And assumed that doveadm was available
   on the same server as it.

   The latest versions support bcrypt and newer hashing algos natively.

   Some tools might rely on the database (mysql/mariadb) to hash 
passwords, so

   this may also be a consideration.

2. Server load / libs:

   - The Dovecot docs: 
https://doc.dovecot.org/configuration_manual/authentication/password_schemes/

 has this to say on ARGON2I/ARGON2ID:

 "Argon2 is the winner of password hashing competition held at July 
2015. The password will
  start with $argon2i$ or $argon2id$. You can use -r to tune 
computational complexity,
  minimum is 3. ARGON2ID is only available if your libsodium is 
recent enough.
  ARGON2 can require quite a hefty amount of virtual memory, so we 
recommend that you set

  service auth { vsz_limit = 2G } at least, or more."

There's a good write up of considering the various algos:

https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html

I considered BLF-CRYPT (for the time being) to be strong enough and a
good balance between compatibility, strength and server load, given the
number of users etc.

Rob


On 2023-06-23 02:14, David Mehler wrote:

Hello,

I'm migrating to a new server. It's running Debian 11 currently though
that's going 12 this weekend. Currently it uses Openssl v3.0.9, and
dovecot 2.3.13 and MySQL (in this case Mariadb) for storing user
account information v10.6.14. My question is in regards password
storage and scheme/encryption/salts.

Currently they are stored in Mariadb password field with a type of
varchar and a 255 character length, and are stored as SHA512-CRYPT.
I'm wondering if I should keep this as is or when I migrate go to
another scheme? I'm thinking argon2i, argon2d, argon2id, sha512,
sha512-crypt, tiger2, saltt?



--
Robert Lister  - email:  r...@lentil.org  - tel: 020 7043 7996
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Server migration

2017-11-27 Thread Joseph Tam



I've asked this before, but now it's time to move one server to
another, I can't delay the operation anymore (the older server is
failing)

Both server are pretty old: 1.2.15

Probably, faster way would be to rsync all mailboxes from the older
server to the newer one.  I can start migrating everything while
running then, stop the older server and sync only what is changed,
keeping downtime at minimum.

Any better solution ?


It depends on your situation.

If you only have a small userbase with modest amount of mail,
you can probably take your whole system down, do a one time
migration, then cut over to the new system without too much
downtime.

If you have medium sized operation, then what you propose above
would work.

If you have a large userbase (or large user mailboxes), or you want to
minimize downtime, you can incrementally migrate per user so that any
particular user experiences a small window of outage, but the system is
online for everybody else.  This, of course, requires a lot more setup
and planning.

Joseph Tam 


Re: Server migration

2017-11-26 Thread Steffen Kaiser

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Fri, 24 Nov 2017, Gandalf Corvotempesta wrote:


I've asked this before, but now it's time to move one server to
another, I can't delay the operation anymore (the older server is
failing)

Both server are pretty old: 1.2.15

Probably, faster way would be to rsync all mailboxes from the older
server to the newer one.
I can start migrating everything while running then, stop the older
server and sync only what is changed, keeping downtime at minimum.

Any better solution ?


No, it would go this way.

- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBWhuqEMQnQQNheMxiAQJxDQf/UHW0IdjQclo81XtGIzs2Wo6L/h6Zw1gd
BBwpS8KaqKSprxOVJY375ybzvwU+POuujmaN2v8TXPRuJY6ptyy57cqfgPPMN1gG
eDp4SoDtQQk0Y1rocM9GdNx5yWb3RLukvpAxLXHaFoQlNRkbIB7kCvNofxiCTcdA
1xcQ7rB1gh+HxCOxf+tLWR/S29EqJeIhxlBUGjTcY42t2hQLBnVwqUJN53GkSWet
h+V10iihSkpd3mXPbc49DV0NWUZTVMuspFNWp74sEeJSaOTYbPQU+im60n93ZWBO
wotPioiQfES561G2+/SOe0ySvG0h92b2ICZWXKRwSRhcCGI4sNdeiw==
=pxDV
-END PGP SIGNATURE-


Re: Server migration

2017-03-20 Thread chaouche yacine
Don't lose any of the dovecot-* files and your clients should be fine. I've 
done 1) a couple of times and nobody got hurt.What you should do is keep the 
two servers (the old and the new one), and once the new one is ready test with 
your client only (change your client's IMAP/POP server settings). Once you make 
sure it works right, you can change the config for the other clients.
I have no idea about 2) 

  -- Yassine.



 

On Monday, March 20, 2017 8:49 AM, Gandalf Corvotempesta 
 wrote:
 

 Hi to all.
It's time to migrate an old server to a newer platform

Some questions:
1) what happens by changing the pop3/IMAP server on the client?
Is the client (Outlook, Thunderbird,...) smart enough to not download every
message again?
I'm asking this because the easier way to migrate would be move all
mailboxes to the new server and then change the hostname on the client

2) what if I add a dovecot proxy on the new server, proxing back all
requests to the older one, if the mailbox is still not migrated?
Would the whole pop3/IMAP transaction happen through the proxy or there is
something an http redirect (or anything similiar to the SIP protocol) ?

3) I think the response to this is no: is dovecot able to log the hostname
used for the connection? I have multiple domains pointing to the same IP.
Something like the Host header in Http.


   


Re: Server migration

2016-11-01 Thread Sami Ketola

> On 1 Nov 2016, at 14.10, Tanstaafl  wrote:
> 
> On 11/1/2016 3:58 AM, Sami Ketola  wrote:
>> On 31 Oct 2016, at 13.11, Tanstaafl  wrote:
>>> Ok, interesting. So... how does dsync do it? Or would it only work
>>> between two dovecot servers?
>>> 
>>> I'm interested in migrating from other servers (Office 365 in one case).
> 
>> Dsync does not use IMAP protocol to store the mails to storage but instead 
>> uses the 
>> dovecot storage API to do that. Internally we can set what ever properties 
>> we want to
>> including IMAP UIDs and POP3 UIDLs. 
>> 
>> Migrating from legacy system should then be done by pulling the mails from 
>> the
>> legacy platform by using the imapc connector and storing them by using the 
>> internal apis.
>> 
>> We can also store mails to imapc: location but in that case there is many 
>> properties that
>> will be lost due to limitations of the IMAP protocol.
> 
> Thanks Sami, but I don't see a definitive answer top my question in the
> above...
> 
> So, when migrating from legacy system (legacy = non-dovecot) using
> imapc, is dovecot able to preserver the UIDs?


If you fetch emails over imapc and store to dovecot dsync will preserve IMAP 
UIDs.

Sami


Re: Server migration

2016-11-01 Thread Tanstaafl
On 11/1/2016 3:58 AM, Sami Ketola  wrote:
> On 31 Oct 2016, at 13.11, Tanstaafl  wrote:
>> Ok, interesting. So... how does dsync do it? Or would it only work
>> between two dovecot servers?
>>
>> I'm interested in migrating from other servers (Office 365 in one case).

> Dsync does not use IMAP protocol to store the mails to storage but instead 
> uses the 
> dovecot storage API to do that. Internally we can set what ever properties we 
> want to
> including IMAP UIDs and POP3 UIDLs. 
> 
> Migrating from legacy system should then be done by pulling the mails from the
> legacy platform by using the imapc connector and storing them by using the 
> internal apis.
> 
> We can also store mails to imapc: location but in that case there is many 
> properties that
> will be lost due to limitations of the IMAP protocol.

Thanks Sami, but I don't see a definitive answer top my question in the
above...

So, when migrating from legacy system (legacy = non-dovecot) using
imapc, is dovecot able to preserver the UIDs?

Thanks, and my apologies for being a bit dense...


Re: Server migration

2016-11-01 Thread Sami Ketola

> On 31 Oct 2016, at 13.11, Tanstaafl  wrote:
> 
> On 10/30/2016 5:32 AM, Sami Ketola  wrote:
>> On 28 Oct 2016, at 16.54, Tanstaafl  wrote:
>>> Oh... I thought the --useuid option eliminated this problem?
>>> 
>>> https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt
> 
>> It does not. There is no option at IMAP level to set the UID.
>> 
>> In this case —useuid seems to keep track on source:uid -> dest:uid
>> pairs on multiple syncs and uses uid numbers to avoid syncing mails
>> as duplicates instead of using headers to do that.
> 
> Ok, interesting. So... how does dsync do it? Or would it only work
> between two dovecot servers?
> 
> I'm interested in migrating from other servers (Office 365 in one case).

Dsync does not use IMAP protocol to store the mails to storage but instead uses 
the 
dovecot storage API to do that. Internally we can set what ever properties we 
want to
including IMAP UIDs and POP3 UIDLs. 

Migrating from legacy system should then be done by pulling the mails from the
legacy platform by using the imapc connector and storing them by using the 
internal apis.

We can also store mails to imapc: location but in that case there is many 
properties that
will be lost due to limitations of the IMAP protocol.

Sami


Re: Server migration

2016-10-31 Thread Tanstaafl
On 10/30/2016 5:32 AM, Sami Ketola  wrote:
> On 28 Oct 2016, at 16.54, Tanstaafl  wrote:
>> Oh... I thought the --useuid option eliminated this problem?
>>
>> https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

> It does not. There is no option at IMAP level to set the UID.
> 
> In this case —useuid seems to keep track on source:uid -> dest:uid
> pairs on multiple syncs and uses uid numbers to avoid syncing mails
> as duplicates instead of using headers to do that.

Ok, interesting. So... how does dsync do it? Or would it only work
between two dovecot servers?

I'm interested in migrating from other servers (Office 365 in one case).

Thanks,

Charles


Re: Server migration

2016-10-30 Thread Sami Ketola

> On 28 Oct 2016, at 16.54, Tanstaafl  wrote:
> 
> On 10/27/2016 8:36 AM, Timo Sirainen  wrote:
>> On 27 Oct 2016, at 15:29, Tanstaafl  wrote:
>>> On 10/26/2016 2:38 AM, Gandalf Corvotempesta
 my only question is: how to manage the email received on the new server
 during the last rsync phase?
>>> 
>>> Use IMAPSync - much better than rsync for this.
> 
>> imapsync will change IMAP UIDs and cause clients to redownload all
>> mails. http://wiki2.dovecot.org/Migration/Dsync should work though.
> 
> Oh... I thought the --useuid option eliminated this problem?
> 
> https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

It does not. There is no option at IMAP level to set the UID.

In this case —useuid seems to keep track on source:uid -> dest:uid pairs on 
multiple syncs
and uses uid numbers to avoid syncing mails as duplicates instead of using 
headers to do that.

Sami


Re: Server migration

2016-10-28 Thread Tanstaafl
On 10/27/2016 8:36 AM, Timo Sirainen  wrote:
> On 27 Oct 2016, at 15:29, Tanstaafl  wrote:
>> On 10/26/2016 2:38 AM, Gandalf Corvotempesta
>>> my only question is: how to manage the email received on the new server
>>> during the last rsync phase?
>>
>> Use IMAPSync - much better than rsync for this.

> imapsync will change IMAP UIDs and cause clients to redownload all
> mails. http://wiki2.dovecot.org/Migration/Dsync should work though.

Oh... I thought the --useuid option eliminated this problem?

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt


Re: Server migration

2016-10-27 Thread Timo Sirainen
On 27 Oct 2016, at 15:58, Gandalf Corvotempesta 
 wrote:
> 
> 2016-10-27 14:36 GMT+02:00 Timo Sirainen :
>> imapsync will change IMAP UIDs and cause clients to redownload all mails. 
>> http://wiki2.dovecot.org/Migration/Dsync should work though.
> 
> Just to be sure: dsync from the *new* node would connect via IMAP to
> the older node and transfer everything ?
> By running this:
> 
> doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc:
> 
> should be OK if newer mails are arrived on the new server ?

Yes.


Re: Server migration

2016-10-27 Thread Gandalf Corvotempesta
2016-10-27 14:36 GMT+02:00 Timo Sirainen :
> imapsync will change IMAP UIDs and cause clients to redownload all mails. 
> http://wiki2.dovecot.org/Migration/Dsync should work though.

Just to be sure: dsync from the *new* node would connect via IMAP to
the older node and transfer everything ?
By running this:

doveadm -o mail_fsync=never sync -1 -R -u user@domain imapc:

should be OK if newer mails are arrived on the new server ?


Re: Server migration

2016-10-27 Thread Timo Sirainen
On 27 Oct 2016, at 15:29, Tanstaafl  wrote:
> 
> On 10/26/2016 2:38 AM, Gandalf Corvotempesta
>  wrote:
>> This is much easier than dovecot replication as i can start immedialy with
>> no need to upgrade the old server
>> 
>> my only question is: how to manage the email received on the new server
>> during the last rsync phase?
> 
> Use IMAPSync - much better than rsync for this.

imapsync will change IMAP UIDs and cause clients to redownload all mails. 
http://wiki2.dovecot.org/Migration/Dsync should work though.


Re: Server migration

2016-10-27 Thread Tanstaafl
On 10/26/2016 2:38 AM, Gandalf Corvotempesta
 wrote:
> This is much easier than dovecot replication as i can start immedialy with
> no need to upgrade the old server
> 
> my only question is: how to manage the email received on the new server
> during the last rsync phase?

Use IMAPSync - much better than rsync for this.


Re: Server migration

2016-10-26 Thread Gandalf Corvotempesta
2016-10-26 8:57 GMT+02:00 Aki Tuomi :
> If you are moving from 1.x to 2.x, I think you should make some trials
> first, and preferably move the user one at a time, blocking access to
> old server/new server during move. It is very forklift upgrade, much danger.

Yes, I'll do some test migration before moving the whole server.
Maildir structure isn't changed between 1.x and 2.x, thus all emails
should be safe.
I have to test the new 2.2 configuration to see if existing users are
able to log-in but how
can I test if existing client would be able to preserve the mail ids
without downloading everything again?


Re: Server migration

2016-10-26 Thread Aki Tuomi


On 26.10.2016 09:38, Gandalf Corvotempesta wrote:
> Il 26 ott 2016 8:30 AM, "Aki Tuomi"  ha scritto:
>> I would recommend using same major release with replication.
>>
>> If you are using maildir++ format, it should be enough to copy all the
>> maildir files over and start dovecot on new server.
>>
> This is much easier than dovecot replication as i can start immedialy with
> no need to upgrade the old server
>
> my only question is: how to manage the email received on the new server
> during the last rsync phase?
> As i wrote previously,  i have some huge maildirs where rsync take hours to
> scan all files
> i can't keep the server down for hours or customers won't receive any new
> emails, so, after the initial sync i have to move the mailbox on the new
> server (only for deliveries) . In this way I'll not loose any emails but
> the new servers as newer data than the old server.
> When doing rsync with --delete, the news mails would be removed
>
> A solution could be to disable customer access to the new server and put
> "new" directory in rsync exclude. Doing this won't delete the newly
> received emails as the "new" directory isn't synced.
> and no one osd able to move from new to cur as users are blocked for login.

If you are moving from 1.x to 2.x, I think you should make some trials
first, and preferably move the user one at a time, blocking access to
old server/new server during move. It is very forklift upgrade, much danger.

Aki


Re: Server migration

2016-10-26 Thread Gandalf Corvotempesta
Il 26 ott 2016 8:30 AM, "Aki Tuomi"  ha scritto:
> I would recommend using same major release with replication.
>
> If you are using maildir++ format, it should be enough to copy all the
> maildir files over and start dovecot on new server.
>

This is much easier than dovecot replication as i can start immedialy with
no need to upgrade the old server

my only question is: how to manage the email received on the new server
during the last rsync phase?
As i wrote previously,  i have some huge maildirs where rsync take hours to
scan all files
i can't keep the server down for hours or customers won't receive any new
emails, so, after the initial sync i have to move the mailbox on the new
server (only for deliveries) . In this way I'll not loose any emails but
the new servers as newer data than the old server.
When doing rsync with --delete, the news mails would be removed

A solution could be to disable customer access to the new server and put
"new" directory in rsync exclude. Doing this won't delete the newly
received emails as the "new" directory isn't synced.
and no one osd able to move from new to cur as users are blocked for login.


Re: Server migration

2016-10-26 Thread Aki Tuomi


On 26.10.2016 09:27, Gandalf Corvotempesta wrote:
> Il 24 ott 2016 5:11 PM, "Michael Seevogel"  ha scritto:
>> I meant your old server. With "old" I was expecting something like Debian
> Sarge or SuSE Linux 9.3. That would have been really old, but since you are
> on Debian Squeeze, I would definitely choose the way with an upgraded
> Dovecot version and its replication service.
>
> Is 2.1 from squeeze-backports enough to start the replication over a newer
> server with dovecot 2.2? Is this supported or both server must run the same
> version?
>
> I've looked around but the replication system is still not clear to me.
> Any howto explaining this in details?

Hi!

I would recommend using same major release with replication.

If you are using maildir++ format, it should be enough to copy all the
maildir files over and start dovecot on new server.

Aki Tuomi
Dovecot oy


Re: Server migration

2016-10-26 Thread Gandalf Corvotempesta
Il 24 ott 2016 5:11 PM, "Michael Seevogel"  ha scritto:
> I meant your old server. With "old" I was expecting something like Debian
Sarge or SuSE Linux 9.3. That would have been really old, but since you are
on Debian Squeeze, I would definitely choose the way with an upgraded
Dovecot version and its replication service.

Is 2.1 from squeeze-backports enough to start the replication over a newer
server with dovecot 2.2? Is this supported or both server must run the same
version?

I've looked around but the replication system is still not clear to me.
Any howto explaining this in details?


Re: Server migration

2016-10-24 Thread Michael Seevogel

Am 24.10.2016 um 15:25 schrieb Gandalf Corvotempesta:

2016-10-24 14:47 GMT+02:00 Michael Seevogel :

If your server OS supports newer Dovecot versions then I would highly
suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and
set up Dovecot's replication[1] feature.


Are you talking about the new server or the older one that I have to replace?
The new server has to be installed from scratch, so, yes, I can use Dovecot 2.2
from Jessie


I meant your old server. With "old" I was expecting something like 
Debian Sarge or SuSE Linux 9.3. That would have been really old, but 
since you are on Debian Squeeze, I would definitely choose the way with 
an upgraded Dovecot version and its replication service.




The "old" server is based on Squeeze, I can upgrade that to Wheezy and install
Dovecot 2.2 from wheezy-backports but I have huge trouble when I've tried to
do the same on a similiar server. I was unable to upgrade the dovecot
configuration
by following the documentation as this didn't work:

doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf

I had an empty  dovecot-2.conf file, no warning or output at all. It
did nothing.



Well, I'am not too familiar with Debian since I'am a Red Hatter but 
perhaps you could use the binaries from there: 
http://wiki2.dovecot.org/PrebuiltBinaries


Dunno if you have to rebuild the binaries, or if you can install them 
straight on Squeeze. You could also try to convert your old dovecot.conf 
on a different machine (maybe your new server?) and then just copy it 
back to your old server.


As a last straw you could certainly adapt the dovecot.conf for Dovecot 
2.2 manually, it shouldn't be too complicated, but this is totally up to 
you.




With this method you can actually archieve a smooth migration while your
current server replicates all emails in real time to your new server,
including new incoming emails and also mailbox changes to your new server
and when the migration is done you'll just have to change your DNS and
disable the Replication service.


Cool.
Any guide about this ?
Should I start the replication on one side and wait for finish before
pointing the mailbox to the new server?



How to setup and start replication is described here: 
http://wiki2.dovecot.org/Replication


Also make sure that you migrate/copy your userdb from the old server to 
the new server and that you properly test the user-mailbox access on the 
new server before you start the replication process.


Regarding replication:
I would wait with adjusting the DNS records until the replication has 
finished and you know that the new server works as expected.


However, you may want to keep the replication process running for one or 
two more days to catch emails still arriving due to DNS caching times on 
your old server. The same may apply to mailusers that still access your 
old server via POP3/IMAP.




Best regards
Michael Seevogel


Re: Server migration

2016-10-24 Thread Gandalf Corvotempesta
2016-10-24 14:47 GMT+02:00 Michael Seevogel :
> P.S. You should think about to use on the new server mdbox as mailbox
> format.
> That's kinda a hybrid of mbox and maildir and benefits of features of both
> its predecessors. However, backup and restoring is in case of mdbox "a bit"
> different. Just have a read...

No, I don't like that format, for this:
This also means that you must not lose the dbox index files, they
can't be regenerated without data loss

additionally, this means to change even our LDA, as neither Exim or
Postfix are able to deliver messages.


Re: Server migration

2016-10-24 Thread Gandalf Corvotempesta
2016-10-24 14:47 GMT+02:00 Michael Seevogel :
> If your server OS supports newer Dovecot versions then I would highly
> suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) and
> set up Dovecot's replication[1] feature.

Are you talking about the new server or the older one that I have to replace?
The new server has to be installed from scratch, so, yes, I can use Dovecot 2.2
from Jessie

The "old" server is based on Squeeze, I can upgrade that to Wheezy and install
Dovecot 2.2 from wheezy-backports but I have huge trouble when I've tried to
do the same on a similiar server. I was unable to upgrade the dovecot
configuration
by following the documentation as this didn't work:

doveconf -n -c /etc/dovecot/dovecot.conf > dovecot-2.conf

I had an empty  dovecot-2.conf file, no warning or output at all. It
did nothing.

> With this method you can actually archieve a smooth migration while your
> current server replicates all emails in real time to your new server,
> including new incoming emails and also mailbox changes to your new server
> and when the migration is done you'll just have to change your DNS and
> disable the Replication service.

Cool.
Any guide about this ?
Should I start the replication on one side and wait for finish before
pointing the mailbox to the new server?

> If you don't want or cannot set up replication you could still do a one-shot
> migration via Dovecot's dsync[2] on the new server, pulling the mails from
> the old. 50GB isn't that much as long as your two servers are at least
> connected with 100 Mbit to the inet. You may want to block for the time of
> the migration via iptables your users accessing Dovecot. However, under the
> bottom-line, if this is really necessary depends on you and the needs of
> your mailusers/customers.

I can't block the whole server. I have to migrate 1 user at once.
But I can disable the pop3/imap access for that user, so noone is
changing the files during the migration
(except for the postfix/exim delivery agent)

> P.S. You should think about to use on the new server mdbox as mailbox
> format.
> That's kinda a hybrid of mbox and maildir and benefits of features of both
> its predecessors. However, backup and restoring is in case of mdbox "a bit"
> different. Just have a read...
>
>
> [1] http://wiki.dovecot.org/Replication
> [2] http://wiki2.dovecot.org/Migration/Dsync

Thank you


Re: Server migration

2016-10-24 Thread Gandalf Corvotempesta
2016-10-24 11:23 GMT+02:00 Karol Augustin :
> When I am doing this I just turn off both servers for the third sync.
> Its short enough to not cause much problem. And then after third sync I
> start the new server and all clients can connect to it so I also
> mitigate any problems resulting from clients that would be still
> connected to the old server. The last issue depends on the way you force
> everyone to use new server (DNS, routing, etc).

The speed for third sync depends on the number of files to be scanned.
I have mailboxes with tons of very small emails, thus even if the first two sync
has transferred all datas, the scan made by rsync to check which files
needs to be transferred
requires many hours.

My own mailbox has 80GB of mails. I can sync everything on a new
server and then start
a new rsync phase. this new phase requires exactly 1 hours and 49
minutes (as I can see from
the last night backup). Transferred data: 78MB. 1 hours and 49 minutes
to transfer only 78MB.

> Remember that beside the new emails that could arrive during sync you
> have also all sorts of user-generated operations as move, delete etc. So
> if you just do 3rd rsync without --delete you can end up duplicating
> users' emails if they move them during procedure.

By shutting down both servers, the "--delete" argument could be used
with no issues.


Re: Server migration

2016-10-24 Thread Michael Seevogel

Am 24.10.2016 um 09:00 schrieb Gandalf Corvotempesta:

Hi
i have to migrate, online, a dovecot 1.2.15 to a new server. Which is the
best way to accomplish this?

I have 2 possibility:
1) migrate from the very old server to a newer server with the same dovecot
version
2) migrate from the very old server to a new server with the latest dovecot
version

can i simply use rsync to sync everything and, when the sync is quick, move
the mailbox from the old server to the new server? My biggest concern is
how to manage the the emails that are coming during the server switch.

Let's assume a 50gb maildir , the first sync would require hours to
complete (tons of very small files) do i can't shutdown the mailbox. The
second sync would require much less time and would also sync the email
received during the first sync (but the mailbox is still receiving new
emails)
now, as third phase, i can move the mailbox to the new server (by changing
the postfix configuration) so that all new emails are received on the new
server and then start the last rsync (by removing the --delete flag or any
new emails would be deleted as not existsnt on the older server)

Any better solution?




If your server OS supports newer Dovecot versions then I would highly 
suggest you to upgrade to Dovecot 2.2.xx (or at least to the latest 2.1) 
and set up Dovecot's replication[1] feature.


With this method you can actually archieve a smooth migration while your 
current server replicates all emails in real time to your new server, 
including new incoming emails and also mailbox changes to your new 
server and when the migration is done you'll just have to change your 
DNS and disable the Replication service.


If you don't want or cannot set up replication you could still do a 
one-shot migration via Dovecot's dsync[2] on the new server, pulling the 
mails from the old. 50GB isn't that much as long as your two servers are 
at least connected with 100 Mbit to the inet. You may want to block for 
the time of the migration via iptables your users accessing Dovecot. 
However, under the bottom-line, if this is really necessary depends on 
you and the needs of your mailusers/customers.




Best regards
Michael Seevogel

P.S. You should think about to use on the new server mdbox as mailbox 
format.
That's kinda a hybrid of mbox and maildir and benefits of features of 
both its predecessors. However, backup and restoring is in case of mdbox 
"a bit" different. Just have a read...



[1] http://wiki.dovecot.org/Replication
[2] http://wiki2.dovecot.org/Migration/Dsync


Re: Server migration

2016-10-24 Thread Karol Augustin
On 2016-10-24 8:00, Gandalf Corvotempesta wrote:

> can i simply use rsync to sync everything and, when the sync is quick, move
> the mailbox from the old server to the new server? My biggest concern is
> how to manage the the emails that are coming during the server switch.
> 
> Let's assume a 50gb maildir , the first sync would require hours to
> complete (tons of very small files) do i can't shutdown the mailbox. The
> second sync would require much less time and would also sync the email
> received during the first sync (but the mailbox is still receiving new
> emails)
> now, as third phase, i can move the mailbox to the new server (by changing
> the postfix configuration) so that all new emails are received on the new
> server and then start the last rsync (by removing the --delete flag or any
> new emails would be deleted as not existsnt on the older server)
> 
> Any better solution?

When I am doing this I just turn off both servers for the third sync.
Its short enough to not cause much problem. And then after third sync I
start the new server and all clients can connect to it so I also
mitigate any problems resulting from clients that would be still
connected to the old server. The last issue depends on the way you force
everyone to use new server (DNS, routing, etc). 

Remember that beside the new emails that could arrive during sync you
have also all sorts of user-generated operations as move, delete etc. So
if you just do 3rd rsync without --delete you can end up duplicating
users' emails if they move them during procedure.


Best,
Karol


[Dovecot] SOLVED (user error) - WAS Re: Server Migration Attempt - new messages DELETED after secondary rsyncs

2013-12-27 Thread Charles Marcus

On 2013-12-27 9:33 AM, Charles Marcus cmar...@media-brokers.com wrote:
Damn! How did I miss that. The 'T' is for 'trashed', so of course, 
when the Inbox is expunged, they will be deleted...


Thanks Rick, for the gentle clue stick... 


Apologies to all for the noise.

--

Best regards,

*/Charles/*