Re: Importing an mbox into Imap
On Sat, 2010-05-29 at 18:34 -0600, Philip Prindeville wrote: > > I'm sure he wouldn't mind sharing the script if you're interested, > but > > it is specific to Cyrus. > > > > poc > > > > > > Ok, just to be clear... does it resubmit the messages via IMAP for > delivery? It runs directly on the Cyrus server, so it probably manipulates the Cyrus file structure directly or calls some internal API, I'm not sure. > Because if it does, it can't be *that* Cyrus-specific... I mean yes, > it can embed flags into the message that are only specific to Cyrus > would take some rewriting, but I can do that... > > So yes, please, share the code if you can. I've asked the author to send it to me or put it on a Web page. I'll let you know. > If I come up with any Dovecot fixes for it, I'll send them back to > you. Sure. poc -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On 5/25/10 9:16 AM, Patrick O'Callaghan wrote: > On Tue, 2010-05-25 at 10:10 +0930, Tim wrote: > >> On Mon, 2010-05-24 at 14:38 -0430, Patrick O'Callaghan wrote: >> >>> Just import the mbox file into any IMAP client (Thunderbird, >>> Evolution, Claws, whatever ...) >>> >> I've tried this with fairly large spool files, on Evolution (painfully >> slow, and needs doing in chunks - select a month's mail, drag and drop >> it between folders), and Thunderbird (eventually fails part way through, >> even in chunks, with no clue as to how far along it got). So I'd like >> to find a good tool for doing this, too. >> > I took the liberty of asking our mail admin about this, since I know > we've done it in the past. This is his answer: > > I wrote a very Cyrus-specific Perl script (which runs on > the mailstore server) that parsed an mbox file, split it into > individual messages, and handed them to Cyrus for storage. And > yes, it takes many hours (on a fast machine with fast disks) and > some babysitting. > > If he's not using Cyrus, or doesn't have access to the server, > tough luck... :-/ > > I'm sure he wouldn't mind sharing the script if you're interested, but > it is specific to Cyrus. > > poc > > Ok, just to be clear... does it resubmit the messages via IMAP for delivery? Because if it does, it can't be *that* Cyrus-specific... I mean yes, it can embed flags into the message that are only specific to Cyrus would take some rewriting, but I can do that... So yes, please, share the code if you can. If I come up with any Dovecot fixes for it, I'll send them back to you. -Philip -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On 05/25/2010 05:41 AM, Tim wrote: >> You may have to build the indexes after the files are moved. > > Hmm, the sort of thing I was hoping to avoid. If I used a mail client > to move the mail, it'd move the messages in a manner that was directly > usable, straight away. Rather than play games directly with the files. Not with dovecot - it is self healing and will auto rebuild indexes as needed. Speed issues are client dependent - e.g if you use thunderbird and leave GLODA turned on .. you're essentially doomed with large mail folders!! Also clients that store mail for offline reading may do it in mbox format (thunderbird for example) and they will simply convert your lovely server into a local mbox .. so turn that off too!!! I had a script somewhere to convert mbox to dovecot maildir ... will dig around and see if i still have it. gene/ -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
Tim wrote: > On Mon, 2010-05-24 at 23:49 -0700, Nataraj wrote: > >> If it's your own mail server and your running an imap server like >> dovecot which uses maildir format, then just use the dovecot script to >> convert your mbox file directly into the file space of the imap >> server. >> > > I am, though don't know the script your referring to. I should probably > have a look at a newer CentOS install, in case it has something that the > older one doesn't. > > >> You may have to build the indexes after the files are moved. >> > > Hmm, the sort of thing I was hoping to avoid. If I used a mail client > to move the mail, it'd move the messages in a manner that was directly > usable, straight away. Rather than play games directly with the files. > >> Another option is to use a local mail client to split the file into >> several smaller folders before trying to move it into imap. >> > > Might give that a go, but I suspect that's still going to be a lot of > time-consuming work. Likewise for taking the first 10,000 lines of text > from a spool, chopping into two files between message sections, and > proceeding that way. > > I would check out the following links and other documentation in the dovecot wiki: http://wiki.dovecot.org/Migration/MailFormat http://wiki.dovecot.org/MailboxFormat My understanding is that the differences in performance between the various formats that use single files has to do with the indexing. This tends to be implementation dependant as much as format related. Though many people consider MH as obsolete, I would consider it as an intermediate format. It will read the maildir file and split it into it's own MH format which uses single files. Even though it can be slow, the nice thing about mh is that it is all command line based and it's easy to write shell scripts, to move and sort messages. Also it tends not to crash. Evolution does read mh format (thunderbird does NOT), and I have used it to move 200mb mh folders into a dovecot server using maildir. I think the key here is to split into smaller folders (maybe even less than 200), even if you put it back into one at the end, that way your less likely to get bitten by evolution crashing 3/4 of the way through. If you get it into 4 folders in dovecot, I believe you can just rename (mv) the files directly on the server (I think the file names are unique) into one folder and reindex the folder. Also, the UW imap server reads mh format, but I have never used it and have no idea what its performance is like. Nataraj -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Tue, 2010-05-25 at 10:46 -0430, Patrick O'Callaghan wrote: > I took the liberty of asking our mail admin about this, since I know > we've done it in the past. This is his answer: > > I wrote a very Cyrus-specific Perl script (which runs on > the mailstore server) that parsed an mbox file, split it into > individual messages, and handed them to Cyrus for storage. And > yes, it takes many hours (on a fast machine with fast disks) and > some babysitting. > > If he's not using Cyrus, or doesn't have access to the server, > tough luck... :-/ > > I'm sure he wouldn't mind sharing the script if you're interested, but > it is specific to Cyrus. I'm using Dovecot. Though, I'm wondering if it will be any advantage because of the difference in server, and: Is there an advantage in externally splitting an mbox spool and playing with files, compared to importing mail from one folder to another, and the mail server storing that mail in maildir, itself. I suspect there's a good chance of creating maildir files that don't fit in with the way the mailserver wants to have its maildir files (file paths, etc.). -- [...@localhost ~]$ uname -r 2.6.27.25-78.2.56.fc9.i686 Don't send private replies to my address, the mailbox is ignored. I read messages from the public lists. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Tue, 2010-05-25 at 10:10 +0930, Tim wrote: > On Mon, 2010-05-24 at 14:38 -0430, Patrick O'Callaghan wrote: > > Just import the mbox file into any IMAP client (Thunderbird, > > Evolution, Claws, whatever ...) > > I've tried this with fairly large spool files, on Evolution (painfully > slow, and needs doing in chunks - select a month's mail, drag and drop > it between folders), and Thunderbird (eventually fails part way through, > even in chunks, with no clue as to how far along it got). So I'd like > to find a good tool for doing this, too. I took the liberty of asking our mail admin about this, since I know we've done it in the past. This is his answer: I wrote a very Cyrus-specific Perl script (which runs on the mailstore server) that parsed an mbox file, split it into individual messages, and handed them to Cyrus for storage. And yes, it takes many hours (on a fast machine with fast disks) and some babysitting. If he's not using Cyrus, or doesn't have access to the server, tough luck... :-/ I'm sure he wouldn't mind sharing the script if you're interested, but it is specific to Cyrus. poc -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Tue, 2010-05-25 at 01:06 -0500, Gabriel Ramirez wrote: > so it's more a year of Fedora mails, in a file, Most definitely. There's a few other big ones, but none are nearly as big as that one. > I has the same problem tens of thousands emails, I evaluate mbox, vs > maildir, decided keep with mbox in separated files, because trying to > rsync that quantity of emails in maildir well seemed in my case take a > long time. A while back I tried comparing mbox to maildir with a staggeringly large number of messages, and found mbox to stay nippy, whereas maildir grinds to a snail's pace, and just gets worse and worse. I'm not trying to keep backup copies, of these, so that issue wouldn't arise for me. > > seems which your only options to separate the mbox are: > > procmail > > but I'm unable to find a procmail recipe to evaluate the original date > field from a email to construct a mailbox in the format > fedora_year_month/ I figured on keeping them all in one folder, rather than separately archiving different periods. So I ought to be able to run some commands that just move *everything* in mbox here over to maildir there? > and will take hours, but will be automatic I do use dovecot. I'm not against it spending ages to complete, so long as it actually does, and doesn't need me to manage it along the way. -- [...@localhost ~]$ uname -r 2.6.27.25-78.2.56.fc9.i686 Don't send private replies to my address, the mailbox is ignored. I read messages from the public lists. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Mon, 2010-05-24 at 23:49 -0700, Nataraj wrote: > If it's your own mail server and your running an imap server like > dovecot which uses maildir format, then just use the dovecot script to > convert your mbox file directly into the file space of the imap > server. I am, though don't know the script your referring to. I should probably have a look at a newer CentOS install, in case it has something that the older one doesn't. > You may have to build the indexes after the files are moved. Hmm, the sort of thing I was hoping to avoid. If I used a mail client to move the mail, it'd move the messages in a manner that was directly usable, straight away. Rather than play games directly with the files. > > Another option is to use a local mail client to split the file into > several smaller folders before trying to move it into imap. Might give that a go, but I suspect that's still going to be a lot of time-consuming work. Likewise for taking the first 10,000 lines of text from a spool, chopping into two files between message sections, and proceeding that way. -- [...@localhost ~]$ uname -r 2.6.27.25-78.2.56.fc9.i686 Don't send private replies to my address, the mailbox is ignored. I read messages from the public lists. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
Tim wrote: > On Mon, 2010-05-24 at 14:38 -0430, Patrick O'Callaghan wrote: > >> Just import the mbox file into any IMAP client (Thunderbird, >> Evolution, Claws, whatever ...) >> > > I've tried this with fairly large spool files, on Evolution (painfully > slow, and needs doing in chunks - select a month's mail, drag and drop > it between folders), and Thunderbird (eventually fails part way through, > even in chunks, with no clue as to how far along it got). So I'd like > to find a good tool for doing this, too. > > -rw--- 1 tim tim 663987436 2010-03-05 18:21 /home/tim/mail/lists/Fedora > > Thanks to the slowness (the above would take hours and hours), I've left > my mail server still running on FC4. The last attempt at moving the > mail, aborted because I simply don't have the time to babysit a machine > for that amount of time, I copied the spool file onto a newly set up > CentOS box as /var/mail/tim and tried dragging the contents of the inbox > to another folder through the IMAP server on the CentOS box, figuring > that that'd be quicker that trying to network it between two mail > servers. It was still excruciatingly painfully slow. > > I want to move from spool files to maildir, because it's getting > impossible to use with huge spool files. Any status change of the spool > file takes ages to complete, and you can't do anything else while you > wait (e.g. you can't read another message). > > If it's your own mail server and your running an imap server like dovecot which uses maildir format, then just use the dovecot script to convert your mbox file directly into the file space of the imap server. Just make sure you understand how the script works and that your putting the files in the right place (I've never used that script). You may have to build the indexes after the files are moved. Another option is to use a local mail client to split the file into several smaller folders before trying to move it into imap. Nataraj -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On 05/24/2010 07:40 PM, Tim wrote: > On Mon, 2010-05-24 at 14:38 -0430, Patrick O'Callaghan wrote: >> Just import the mbox file into any IMAP client (Thunderbird, >> Evolution, Claws, whatever ...) > > I've tried this with fairly large spool files, on Evolution (painfully > slow, and needs doing in chunks - select a month's mail, drag and drop > it between folders), and Thunderbird (eventually fails part way through, > even in chunks, with no clue as to how far along it got). So I'd like > to find a good tool for doing this, too. > > -rw--- 1 tim tim 663987436 2010-03-05 18:21 /home/tim/mail/lists/Fedora Hi, so it's more a year of Fedora mails, in a file, I has the same problem tens of thousands emails, I evaluate mbox, vs maildir, decided keep with mbox in separated files, because trying to rsync that quantity of emails in maildir well seemed in my case take a long time. seems which your only options to separate the mbox are: procmail but I'm unable to find a procmail recipe to evaluate the original date field from a email to construct a mailbox in the format fedora_year_month/ deliver from dovecot using a sieve script, that I have it working (I have the sieve rules to evaluate the date from the email but I need to extract the relevant parts from the script because filter many things more which mailing lists and is only tested with deliver from dovecot, so if you don't use dovecot will not work for you and will take hours, but will be automatic Gabriel -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Mon, 2010-05-24 at 14:38 -0430, Patrick O'Callaghan wrote: > Just import the mbox file into any IMAP client (Thunderbird, > Evolution, Claws, whatever ...) I've tried this with fairly large spool files, on Evolution (painfully slow, and needs doing in chunks - select a month's mail, drag and drop it between folders), and Thunderbird (eventually fails part way through, even in chunks, with no clue as to how far along it got). So I'd like to find a good tool for doing this, too. -rw--- 1 tim tim 663987436 2010-03-05 18:21 /home/tim/mail/lists/Fedora Thanks to the slowness (the above would take hours and hours), I've left my mail server still running on FC4. The last attempt at moving the mail, aborted because I simply don't have the time to babysit a machine for that amount of time, I copied the spool file onto a newly set up CentOS box as /var/mail/tim and tried dragging the contents of the inbox to another folder through the IMAP server on the CentOS box, figuring that that'd be quicker that trying to network it between two mail servers. It was still excruciatingly painfully slow. I want to move from spool files to maildir, because it's getting impossible to use with huge spool files. Any status change of the spool file takes ages to complete, and you can't do anything else while you wait (e.g. you can't read another message). -- [...@localhost ~]$ uname -r 2.6.27.25-78.2.56.fc9.i686 Don't send private replies to my address, the mailbox is ignored. I read messages from the public lists. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Mon, 2010-05-24 at 11:59 -0600, Philip Prindeville wrote: > I have an archived rmail-style "mbox" that I want to move into my Imap > message store. > > Anyone have a Perl script that will read an mbox, split it into > individual messages, and then redeliver those messages via an Imap > connection? Just import the mbox file into any IMAP client (Thunderbird, Evolution, Claws, whatever ...) poc -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
Frantisek Hanzlik wrote: > Philip Prindeville wrote: >> I have an archived rmail-style "mbox" that I want to move into my Imap >> message store. >> >> Anyone have a Perl script that will read an mbox, split it into >> individual messages, and then redeliver those messages via an Imap >> connection? >> >> Thanks, >> >> -Philip > > In dovecot RPM package is perl script "perfectmaildir.pl" which do this > conversion. > > Franta Hanzlik I was too hurry, "perfectmaildir.pl" places mails directly to user maildir directory, not over IMAP connection. Sorry. FH -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
On Mon, 24 May 2010 11:59:48 -0600 Philip Prindeville wrote: > Anyone have a Perl script that will read an mbox, split it into > individual messages, and then redeliver those messages via an Imap > connection? I don't know about perl, but I clearly remember importing giant mbox files into my imap server using claws-mail. It wouldn't surprise me if almost every mail client had an "import" function somewhere in the menus. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Re: Importing an mbox into Imap
Philip Prindeville wrote: > I have an archived rmail-style "mbox" that I want to move into my Imap > message store. > > Anyone have a Perl script that will read an mbox, split it into > individual messages, and then redeliver those messages via an Imap > connection? > > Thanks, > > -Philip In dovecot RPM package is perl script "perfectmaildir.pl" which do this conversion. Franta Hanzlik -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Importing an mbox into Imap
I have an archived rmail-style "mbox" that I want to move into my Imap message store. Anyone have a Perl script that will read an mbox, split it into individual messages, and then redeliver those messages via an Imap connection? Thanks, -Philip -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
