Re: using Murder for migration from UW IMAP
On Monday 09 December 2002 23:10 pm, John A. Tamplin wrote: Quoting Jeremy Rumpf [EMAIL PROTECTED]: You might also want to consider using the perdition IMAP/Pop3 proxy. It's well suited for something like this since it can use LDAP, MySQL, PostreSQL, and local GDBM databases to lookup the real server for users. http://www.vergenet.net/linux/perdition/ Thanks for the pointer. Do you have a suggestion for an LMTP proxy that might use the same back-end database? Not off hand, you can use Postfix's transport maps to kind of fake being an LMTP proxy. That would require you to put another MTA hop in the mix, or replace your boarder MTAs with postfix. I'm not sure how your layout is, but postfix + LDAP can do some pretty nifty things when it comes to mail routing (I'm sure other MTAs can as well, but I'm mostly fluent with postfix). From postfix's perspective, one address could be delivered via LMTP to a Cyrus mail backend, while another address can be SMTP delievered to another MTA running on a UW IMAP backend. Routing can be switched on the fly by updating the LDAP entry. I kind of describe what I'm talking about in this post: http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrussearchterm=perditionmsg=17997 I have plans to wrap this up into a more formal howto, but haven't found the time yet. If you have any other questions/ideas, feel free to ask. Cheers, Jeremy
Re: using Murder for migration from UW IMAP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Content-Type: text/plain; charset=us-ascii In message [EMAIL PROTECTED], John A. Tamplin writ es: Quoting Rob Siemborski [EMAIL PROTECTED]: Do you really have so much data that it would take 2-3 days to move it? The tests I have done so far (using both imap-utils mbxcvt and custom perl code) it looks like it will take around 60 hours to convert everything (2300 users, 23000 folders, 70G total), and that is putting /var/imap on a tmpfs filesystem for the conversion (about a 2x speed improvement there) and striping /cyrus across 14 drives in an FC-AL array. Have you considered just doing a direct mailbox conversion? You can convert directly to Cyrus mailboxes and then reconstruct everything. That might be faster than dealing with the mbxcvt code. - -- Ted Cabeen http://www.pobox.com/~secabeen[EMAIL PROTECTED] Check Website or Keyserver for PGP/GPG Key BA0349D2 [EMAIL PROTECTED] I have taken all knowledge to be my province. -F. Bacon [EMAIL PROTECTED] Human kind cannot bear very much reality.-T.S.Eliot[EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.1 (FreeBSD) Comment: Exmh version 2.5 07/13/2001 iD8DBQE99lFSoayJfLoDSdIRAp5iAKDJGxcgRfjU6k1QI3zWkIzKrOcVRQCeIKBP IqjyY9I+VrxVbx3pkHu8DiU= =dvwG -END PGP SIGNATURE-
Re: using Murder for migration from UW IMAP
Ted Cabeen wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Content-Type: text/plain; charset=us-ascii In message [EMAIL PROTECTED], John A. Tamplin writ es: Quoting Rob Siemborski [EMAIL PROTECTED]: Do you really have so much data that it would take 2-3 days to move it? The tests I have done so far (using both imap-utils mbxcvt and custom perl code) it looks like it will take around 60 hours to convert everything (2300 users, 23000 folders, 70G total), and that is putting /var/imap on a tmpfs filesystem for the conversion (about a 2x speed improvement there) and striping /cyrus across 14 drives in an FC-AL array. Have you considered just doing a direct mailbox conversion? You can convert directly to Cyrus mailboxes and then reconstruct everything. That might be faster than dealing with the mbxcvt code. I need to get the flags and seen state transferred as well -- that wouldn't do it, right? It is important that the transition be as seemless as possible. Having to rename a few mailboxes that UW IMAP allows but Cyrus doesn't is enough hassle to the users. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
using Murder for migration from UW IMAP
Would it be possible to use Murder to migrate from UW IMAP? I have Cyrus setup and running on a new machine, but the problem is that taking everything down and converting all the mailboxes would be too much downtime (2-3 days). What I was thinking of is setting up a frontend server with UW IMAP and Cyrus as the backend servers, with the initial mupdated database showing all the mailboxes were on the old IMAP server. UW IMAP isn't going to talk to mupdated, but if no mailboxes are being created there it seems like that won't matter. As each mailbox is moved to the new server, the master list of mailboxes is reflected to update the new location. From my inspection of the documentation and some of the code, it looks like this should work with a couple of caveats (during the transition, users have to be prevented from creating new mailboxes -- this can be hacked in the code, and connections that have been referred to the old server have to be broken when the mailbox is transferred). Does anyone see a reason why this won't work or any other gotchas to watch for? Thanks. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: using Murder for migration from UW IMAP
On Mon, 9 Dec 2002, John Alton Tamplin wrote: Would it be possible to use Murder to migrate from UW IMAP? I have Cyrus setup and running on a new machine, but the problem is that taking everything down and converting all the mailboxes would be too much downtime (2-3 days). What I was thinking of is setting up a frontend server with UW IMAP and Cyrus as the backend servers, with the initial mupdated database showing all the mailboxes were on the old IMAP server. UW IMAP isn't going to talk to mupdated, but if no mailboxes are being created there it seems like that won't matter. As each mailbox is moved to the new server, the master list of mailboxes is reflected to update the new location. From my inspection of the documentation and some of the code, it looks like this should work with a couple of caveats (during the transition, users have to be prevented from creating new mailboxes -- this can be hacked in the code, and connections that have been referred to the old server have to be broken when the mailbox is transferred). Does anyone see a reason why this won't work or any other gotchas to watch for? Thanks. So, the murder really isn't intended to work like this. Does UW-IMAPd even support proxy authentication? If it does, *offhand*, provided you're not using internal mechanisms to move the mailboxes (i.e. you are moving them by hand), I don't really see a reason this won't work, but I'm not going to recommend it. Do you really have so much data that it would take 2-3 days to move it? -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: using Murder for migration from UW IMAP
Quoting Rob Siemborski [EMAIL PROTECTED]:
So, the murder really isn't intended to work like this.
Right, I understand. I also looked for more generic IMAP/LMTP proxies but most
of the links were dead and the ones that weren't seemed to be lacking something.
Does UW-IMAPd even support proxy authentication?
No, logins to it will have to be the user who is accessing their mailboxes.
Once the client logs into the proxy, isn't that same login information passed
onto the backend server?
If it does, *offhand*, provided you're not using internal mechanisms to
move the mailboxes (i.e. you are moving them by hand), I don't really see
a reason this won't work, but I'm not going to recommend it.
Here is pseudo-code of how I envisioned the process:
Populate mupdated with initial database showing all mailboxes on old server
for each user {
block mail delivery to the user (LMTP returns temporary failure)
block write access in UW IMAP (minor hack to locking code)
connect to Cyrus server as user with backdoor password (via saslauthd)
for each user mailbox {
create mailbox on Cyrus server
Do you really have so much data that it would take 2-3 days to move it?
The tests I have done so far (using both imap-utils mbxcvt and custom perl code)
it looks like it will take around 60 hours to convert everything (2300 users,
23000 folders, 70G total), and that is putting /var/imap on a tmpfs filesystem
for the conversion (about a 2x speed improvement there) and striping /cyrus
across 14 drives in an FC-AL array.
--
John A. Tamplin
Unix System Administrator
Re: using Murder for migration from UW IMAP
On Mon, 9 Dec 2002, John A. Tamplin wrote: No, logins to it will have to be the user who is accessing their mailboxes. Once the client logs into the proxy, isn't that same login information passed onto the backend server? No, the IMAP proxies use a superuser account to auth to the backends. It's not quite an IMAP admin, but its pretty close. This is because mechanisms such as KERBEROS_V4 and GSSAPI make passing credentials hard. Do you really have so much data that it would take 2-3 days to move it? The tests I have done so far (using both imap-utils mbxcvt and custom perl code) it looks like it will take around 60 hours to convert everything (2300 users, 23000 folders, 70G total), and that is putting /var/imap on a tmpfs filesystem for the conversion (about a 2x speed improvement there) and striping /cyrus across 14 drives in an FC-AL array. Perhaps you'd be better off using Perdition or another IMAP proxy to do the move, since the murder is really trying to solve a slightly different problem. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: using Murder for migration from UW IMAP
On Monday 09 December 2002 06:37 pm, John Alton Tamplin wrote: Would it be possible to use Murder to migrate from UW IMAP? I have Cyrus setup and running on a new machine, but the problem is that taking everything down and converting all the mailboxes would be too much downtime (2-3 days). What I was thinking of is setting up a frontend server with UW IMAP and Cyrus as the backend servers, with the initial mupdated database showing all the mailboxes were on the old IMAP server. UW IMAP isn't going to talk to mupdated, but if no mailboxes are being created there it seems like that won't matter. As each mailbox is moved to the new server, the master list of mailboxes is reflected to update the new location. From my inspection of the documentation and some of the code, it looks like this should work with a couple of caveats (during the transition, users have to be prevented from creating new mailboxes -- this can be hacked in the code, and connections that have been referred to the old server have to be broken when the mailbox is transferred). Does anyone see a reason why this won't work or any other gotchas to watch for? Thanks. You might also want to consider using the perdition IMAP/Pop3 proxy. It's well suited for something like this since it can use LDAP, MySQL, PostreSQL, and local GDBM databases to lookup the real server for users. http://www.vergenet.net/linux/perdition/ Cheers, Jeremy
Re: using Murder for migration from UW IMAP
Quoting Jeremy Rumpf [EMAIL PROTECTED]: You might also want to consider using the perdition IMAP/Pop3 proxy. It's well suited for something like this since it can use LDAP, MySQL, PostreSQL, and local GDBM databases to lookup the real server for users. http://www.vergenet.net/linux/perdition/ Thanks for the pointer. Do you have a suggestion for an LMTP proxy that might use the same back-end database? -- John A. Tamplin Unix System Administrator
