Re: Bare newlines problem
Joseph Brennan schreef: --On Tuesday, April 3, 2007 13:27 +0200 Paul van der Vlis [EMAIL PROTECTED] wrote: Hello, When I move a message to another mailbox, I get a warning about bare newlines. How can I remove these bare newlines? It's a big message with foto's, 3.5 MB. I am not sure this warning is correct. I allready tried a perl-script of Joseph Brennan what I found in this list, but it did not change the message (checked with diff). The script prepares mbox-format mailboxes for mailutil, which complains if there are CR characters (\015) in the mbox-format files. It changes CRLF to LF and then changes remaining CR to LF. Bare LF is normal for unix files. Are you sure it says bare newlines? Newline is an ambiguous term. Yes. The warning is: Message contains bare newlines. It's Cyrus 2.1. I hope somebody can tell me how I can remove the bare newline(s) in the message. With regards, Paul van der Vlis. Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- http://www.vandervlis.nl/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Bare newlines problem
Paul van der Vlis wrote: It's a big message with foto's, 3.5 MB. I am not sure this warning is correct. I hope somebody can tell me how I can remove the bare newline(s) in the message. I've had to deal with this issue when moving such a message between accounts, in my case from a UW-IMAP server using mbx to a Cyrus 2.3 server. While forwarding the message usually works, it's inelegant. Oddly, I've found that simply copying it to temporary folder on the UW-IMAP server before copying it to Cyrus is often all that's needed for the operation to be successful. YMMV. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Bare newlines problem
Hello,
When I move a message to another mailbox, I get a warning about bare
newlines. How can I remove these bare newlines?
It's a big message with foto's, 3.5 MB. I am not sure this warning is
correct.
I allready tried a perl-script of Joseph Brennan what I found in this
list, but it did not change the message (checked with diff).
--
while() {
# The \000 character (NUL) is not allowed
if ($line =~ s/\000//g) {
print STDERR WARNING: Removing NUL\n;
}
# Change CRLF or bare CR to LF
$endcr = $midcr = 0;
$endcr++ if ($line =~ s/\015$//g); # \n already there
$midcr++ if ($line =~ s/\015/\n/g); # add \n
if ($endcr || $midcr) {
print STDERR WARNING: Correcting CR characters\n;
}
print;
}
-
With regards,
Paul van der Vlis.
--
http://www.vandervlis.nl/
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Bare newlines problem
--On Tuesday, April 3, 2007 13:27 +0200 Paul van der Vlis [EMAIL PROTECTED] wrote: Hello, When I move a message to another mailbox, I get a warning about bare newlines. How can I remove these bare newlines? It's a big message with foto's, 3.5 MB. I am not sure this warning is correct. I allready tried a perl-script of Joseph Brennan what I found in this list, but it did not change the message (checked with diff). The script prepares mbox-format mailboxes for mailutil, which complains if there are CR characters (\015) in the mbox-format files. It changes CRLF to LF and then changes remaining CR to LF. Bare LF is normal for unix files. Are you sure it says bare newlines? Newline is an ambiguous term. Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: A script for fixing bare newlines in mailbox files?
--On Thursday, January 11, 2007 17:35 -0500 Zachariah Mully
[EMAIL PROTECTED] wrote:
Howdy all-
We've been bitten by migrating some of our people from Outlook to
Thunderbird, and then using Tbird to move their mail off their local
machines onto the IMAP server where it belongs. Unfortunately we've not
patched Cyrus to accept bare newlines, nor intend to... Since I have
access to the local mailboxes does anybody have a perl script or
something of the like that would remove the bare newlines from the raw
mailbox files? My perl-fu sucks this days, and I've not been able to
figure where and how to remove them...
When moving from U Wash to Cyrus we applied this rewrite to all
mailboxes. Get rid of any nulls while you're at it.
while() {
# The \000 character (NUL) is not allowed
if ($line =~ s/\000//g) {
print STDERR WARNING: Removing NUL\n;
}
# Change CRLF or bare CR to LF
$endcr = $midcr = 0;
$endcr++ if ($line =~ s/\015$//g); # \n already there
$midcr++ if ($line =~ s/\015/\n/g); # add \n
if ($endcr || $midcr) {
print STDERR WARNING: Correcting CR characters\n;
}
print;
}
Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: A script for fixing bare newlines in mailbox files?
Joseph Brennan wrote:
When moving from U Wash to Cyrus we applied this rewrite to all
mailboxes. Get rid of any nulls while you're at it.
while() {
# The \000 character (NUL) is not allowed
if ($line =~ s/\000//g) {
print STDERR WARNING: Removing NUL\n;
}
# Change CRLF or bare CR to LF
$endcr = $midcr = 0;
$endcr++ if ($line =~ s/\015$//g); # \n already there
$midcr++ if ($line =~ s/\015/\n/g); # add \n
if ($endcr || $midcr) {
print STDERR WARNING: Correcting CR characters\n;
}
print;
}
Did any users report any further corruption of what is arguably already
a corrupted message? I'm not familiar with the cause of this problem,
but having encountered it before, mainly with messages that have large
attachments, I'm wondering if attached files might be unusable after
such a scrubbing (assuming they were not encoded properly).
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: A script for fixing bare newlines in mailbox files?
--On Friday, January 12, 2007 12:18 -0500 Jorey Bump [EMAIL PROTECTED] wrote: Did any users report any further corruption of what is arguably already a corrupted message? I'm not familiar with the cause of this problem, but having encountered it before, mainly with messages that have large attachments, I'm wondering if attached files might be unusable after such a scrubbing (assuming they were not encoded properly). No. For a few years we have been refusing or rewriting messages with nulls or bare returns, so only mail at least 4 years old was involved. Many of the messages involved were 10 years old or more. Amateurish Windows-based mail-sending software is still in use that sends junk like this. From the lack of trouble reports, I think it is text parts that are mainly affected. Maybe to do encoding the software writers use standard modules that do it right. Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
A script for fixing bare newlines in mailbox files?
Howdy all- We've been bitten by migrating some of our people from Outlook to Thunderbird, and then using Tbird to move their mail off their local machines onto the IMAP server where it belongs. Unfortunately we've not patched Cyrus to accept bare newlines, nor intend to... Since I have access to the local mailboxes does anybody have a perl script or something of the like that would remove the bare newlines from the raw mailbox files? My perl-fu sucks this days, and I've not been able to figure where and how to remove them... Thanks, Z Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: A script for fixing bare newlines in mailbox files?
On Thursday 11 January 2007 23:35, Zachariah Mully wrote: We've been bitten by migrating some of our people from Outlook to Thunderbird, and then using Tbird to move their mail off their local machines onto the IMAP server where it belongs. Unfortunately we've not patched Cyrus to accept bare newlines, nor intend to... Since I have access to the local mailboxes does anybody have a perl script or something of the like that would remove the bare newlines from the raw mailbox files? My perl-fu sucks this days, and I've not been able to figure where and how to remove them... What do you mean with bare newlines in mailboxes.db? What does an Export as Textfile show? (as cyrus) $ ctl_mboxlist -d -- Andreas Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: A script for fixing bare newlines in mailbox files?
Andreas Winkelmann wrote: What do you mean with bare newlines in mailboxes.db? What does an Export as Textfile show? (as cyrus) $ ctl_mboxlist -d Sorry, I meant the raw mailboxes used by Tbird on their own machines. Because they were originally using Outlook, many people inadvertently saved their mail on their local machines not on the server. We've moved them to Tbird which does a good job of importing the local mail from Outlook, but now we can't move the mailboxes to the server because they contain barenewlines. So short of patching Cyrus to accept these messages (which I'll only do as a last resort), I was wondering if anybody had a nice little piece of perl that I could let loose on these mailboxes to scrub the bare newlines from them. Thanks, Z Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Bare newlines problem
Hi, I'm doing a large import of mail from Outlook and into Cyrus. My problem is with emails that contain bare newlines. Cyrus rejects these mails (as it probably should). What I am wondering about, is if anyone has made a patch so that cyrus may fix the mail or at least let it pass into the messagestore ? Kind regards, Tarjei --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Bare newlines problem
Hi, I end up answering myself here :)
On Sat, 2005-07-09 at 16:17 +0200, [EMAIL PROTECTED] wrote:
Hi, I'm doing a large import of mail from Outlook and into Cyrus. My problem
is
with emails that contain bare newlines.
Cyrus rejects these mails (as it probably should).
What I am wondering about, is if anyone has made a patch so that cyrus may fix
the mail or at least let it pass into the messagestore ?
Fedora Core 3 included a Cyrus version with a nobarenewlines patch.
I've included it here so that the next person who comes by should be
able to use it:
--- cyrus-imapd-2.1.15/imap/message.c.orig 2003-11-11
15:45:11.0 +0100
+++ cyrus-imapd-2.1.15/imap/message.c 2003-11-11 15:35:11.0
+0100
@@ -246,7 +246,8 @@
for (p = (unsigned char *)buf; *p; p++) {
if (*p == '\n') {
- if (!sawcr) r = IMAP_MESSAGE_CONTAINSNL;
+ /* Do *NOT* check for RFC compliant line breaks (bare
newlines) */
+ /* if (!sawcr) r = IMAP_MESSAGE_CONTAINSNL; */
sawcr = 0;
if (blankline) {
inheader = 0;
If someone has a more elegant solution, I'm all ears :)
Regards,
Tarjei
Kind regards,
Tarjei
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
Tarjei Huse [EMAIL PROTECTED]
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Bare newlines problem
Hi all, I have a problem with moving mails from Netscape to a Cyrus (1.5.19) mail server. There a several mails which are in wrong format and Cyrus complains about bare newlines. Is it possible to deactivate this check? I know that Cyrus is right to complain about these mails, but I need them to move to my new server. ;-) Thanks, Roland
Bare newlines
I was attempting to move an email message from a local account onto an IMAP directory located on Cyrus. The client I am using is Netscape Mail and usually it gives me no problems. This time, I am not able to move the message because the client gives the following error: Unable to perform operation. Message contains bare newlines. The message has an attached PDF file (2Mb size) and a signature with image integrated. Any idea of what can be my problem? TIA - En3pY -- Sebastian Konstanty Zdrojewski IT Analyst Neticon S.r.l. via Valtellina, 16 - 20159 Milano Tel. +39 02 68.80.731 FAX +39 02.60.85.70.41 Cell. +39 349.33.04.311 ICQ # 97334916 -- Web: http://www.neticon.it/ E-mail: [EMAIL PROTECTED]
Re: Bare newlines
Sebastian Konstanty Zdrojewski wrote: I was attempting to move an email message from a local account onto an IMAP directory located on Cyrus. The client I am using is Netscape Mail and usually it gives me no problems. This time, I am not able to move the message because the client gives the following error: Unable to perform operation. Message contains bare newlines. The message has an attached PDF file (2Mb size) and a signature with image integrated. Any idea of what can be my problem? In IMAP, messages are required to have \r\n as line terminators, not just \n. Many other IMAP servers (or mail clients' local folders) do not enforce strict RFC compliance, while Cyrus does -- when we converted from UW-IMAP to Cyrus I had to write code to munge messages into the proper format (8 bit or null data in headers, and bogus headers are the other two issues you will likely run into). -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Bare newlines
On Fri, 28 Mar 2003, John Alton Tamplin wrote: Sebastian Konstanty Zdrojewski wrote: I was attempting to move an email message from a local account onto an IMAP directory located on Cyrus. The client I am using is Netscape Mail and usually it gives me no problems. This time, I am not able to move the message because the client gives the following error: Unable to perform operation. Message contains bare newlines. The message has an attached PDF file (2Mb size) and a signature with image integrated. Any idea of what can be my problem? In IMAP, messages are required to have \r\n as line terminators, not just \n. Many other IMAP servers (or mail clients' local folders) do not enforce strict RFC compliance, while Cyrus does -- when we converted from UW-IMAP to Cyrus I had to write code to munge messages into the proper format (8 bit or null data in headers, and bogus headers are the other two issues you will likely run into). John, How are you dealing with 8 bit or null data in headers, and bogus headers? - Bill -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Message contains bare newlines
hi everybody, I have some mail no delivered for this following reason. I don't know what does it mean Message contains bare newlines. Have someone got an idea ? Many thks fr yr help. Jean-Marc Forcer la livraison des courriers avec la commande '/usr/lib/sendmail -v -q' Running /var/spool/mqueue/f6AHS3v02637 (sequence 1 of 8) [EMAIL PROTECTED]... Connecting to /var/imap/socket/lmtp via cyrus... 220 lituf1.tuf.telintrans.fr LMTP Cyrus v2.0.11 ready LHLO lituf1.tuf.telintrans.fr 250-lituf1.tuf.telintrans.fr 250-8BITMIME 250-IGNOREQUOTA 250-ENHANCEDSTATUSCODES 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5 EXTERNAL 250 PIPELINING MAIL From:[EMAIL PROTECTED] 250 2.1.0 ok RCPT To:deumier_s 250 2.1.5 ok DATA 354 go ahead . 554 5.6.0 Message contains bare newlines [EMAIL PROTECTED]... Service unavailable RSET 250 2.0.0 ok
bare newlines
What exactly are "bare newlines" and has anyone a strategy for getting rid of them easily so that deliver doesn't freak out and create irritating mail loops every time we get a malformed e-mail from Oracle's cron? TIA, -danny -- http://dannyman.toldme.com/
Re: 'message contains bare newlines' - was Re: cyrus imapd 1.6.25 beta version
"David L. Parsley" wrote: I've tried to dig through the archives a few times, but haven't found good info on the problems people were having with 'message contains bare newlines'. I get several of these a week from the linux kernel mailing list. Messages like this get abandoned in /var/spool/mqueue; i.e., 'mailq' doesn't list 'em, but 'ls /var/spool/mqueue' sure does! Essentially, it seems like cyrus+sendmail just lets these messages drop on the floor - yech! Any clue whether this behavior was fixed in .25beta? I took a quick look at the code; nothing in the vicinity of the bare-newline check appeared to have changed. So I just put in an (UGLY) little hack, which munges a bare newline to an 'X'. I guess it would also be easy to make cyrus just ignore these, but I thought an X might stand out so I'd see where it's occuring. (not yet) I remember people saying bare newlines violated RFC's, but shouldn't cyrus be as lenient as possible in what it accepts, and as compliant as possible in what it sends/produces? I didn't do anything, nor do I know of anything that was done regarding bare newlines. I'll defer to Larry on this one. Ken -- Kenneth Murchison Oceana Matrix Ltd. Software Engineer 21 Princeton Place 716-662-8973 x26 Orchard Park, NY 14127 --PGP Public Key--http://www.oceana.com/~ken/ksm.pgp
