Robert Burrell Donkin ha scritto:
On Sun, Jul 20, 2008 at 9:08 PM, Stefano Bagnara <[EMAIL PROTECTED]> wrote:
Robert Burrell Donkin ha scritto:
On Sat, Jul 19, 2008 at 5:29 PM, Stefano Bagnara <[EMAIL PROTECTED]> wrote:
Oleg Kalnichevski ha scritto:
Robert Burrell Donkin wrote:
On 7/18/08, Stefano Bagnara <[EMAIL PROTECTED]> wrote:
Robert Burrell Donkin ha scritto:
On Fri, Jul 18, 2008 at 9:34 AM, Stefano Bagnara <[EMAIL PROTECTED]>
wrote:
Robert Burrell Donkin ha scritto:
<snip>
2) ((TextBody) b).getReader(). This give me a reader, so this support
the "line" concept: I do expect this one to treat "non canonical"
newlines like the header/structure parser: if headers are allowed to
terminate with an isolated LF then also lines in text content should
do
the same (because probably the whole mime message has LF instead of
CRLF). [RFC seems to suggest that the fact is that the MIME message is
encoded using LF instead of CRLF and that this specific encoding
breaks
binary parts, but we want to be smarter wrt this issue].
TextBody is part of the DOM. This can and should be addressed there
(rather than in the parser). I think that doing this should satisfy
both needs without compromising the performance of the parser.
If this is indeed something we can all agree on, I can try to solve the
first problem (strict/lenient line delimiter handling) using a pluggable
strategy of some kind.
Oleg
My limited knowledge of mime4j details doesn't let me reply "+1". So I
simply tell what I expect from mime4j as an user:
it's important to understand that mime4j targets different kinds of
user. the pull parser is a low level application agnostic interface
aimed at experts who need performance. the DOM and SAX components are
higher level interfaces for less experience users who are willing to
compromise flexibility and performance. each user will have different
expectations.
Lenient line delimiter parsing:
- consider isolated LF and CR in the mime stream as newlines as long as a
newline concept exists in that specific place (everywhere but binary body
parts having ContentTransferEncoding = "binary").
the low level interface should allow the user to determine whether
they want to canonicalise. the higher level interface should probably
canonicalise.
I have an alternative proposal, see the bottom of this message.
- This means that a CR in a base64 stream is a newline, a CR in a
text/plain
is a newline, a "CR<boundary> CR" sequence is a valid multipart boundary,
"CRLFCR", "CRLFCR", "CRCRLF", "LFCRLF", "LFCR", "CRCR" or "LFLF"
sequences
are valid separators between header and body because they are considered
as
equivalent to "CRLFCRLF".
i'm not sure i agree (i need to think about this a little more)
Ok, let me know your doubts as you get them.
- THis also means that writing in output this stuff will result in a mime
stream with NO isolated CRs or LFs (unless they are in a "binary" encoded
body).
i'm happy for the high level DOM API to perform conversions on the
streams.
Strict line delimiter parsing (I don't care if we have this now, I just
think we should have this in mind while factoring mime4j because it
should
be possible to implement this with no major changes).
this is a non-goal as far as i'm concerned. performant validating
parsers tend to be more difficult to write. if a validating engine is
needed then i'd prefer to approach the design without preconditions. i
need a fast robust parser that is able to cope with practical MIME
documents whether they are valid or not.
Ok, no one seems to care about strict parsing, so let's forget about this
for now, but please let me understand this:
I see mime4j already have a strict parsing concept about throwing exceptions
vs monitor calls when it encounter malformed/unexpected content: what is the
rationale for needing the current strict parsing while not needing the CRLF
delimiter strict parsing?
- LFs and CRs are not newlines, they are not considered newlines and
results
in errors raised by the parser (invalid header, invalid content, and so
on)
that will result in a parsing failure or (if the raised errors are
ignored)
in invalid DOM (I'm not sure how we currently handle this case for
non-expected 8bit content in an header, but it should be the same).
- writing in output this content should result in a well-formed content,
so:
- if an LF in the header is somehow "encodable" as a valid sequence it
should be parsed as LF and then encoded while outputting. If instead an
LF
in the header is not encodable then we should fail parsing or remove it
(or
convert it to "?" or anything similar) if we want to be lenient.
i'm happy for this to be added to the high level DOM
I'm not saying that I want mime4j to support all of this before a
release, I
just want to understand if this is what you also expect and if this can
be
considered a common goal.
i'm happy to address your concerns by adding conversion code into the
higher level API layers but if mime4j seriously needs to compromise
the low level API then i'm not sure i can use this library for my mail
work either. in this case, i'd be happy to introduce a proposal for a
performant low level pull parser for MIME to the commons instead.
I'm working on a solution having readLine methods not returning the newline
chars so that the user of readLine does not need to care about line
delimiter.
This way we can tune the line delimiter inside the
BufferedLineReaderInputStream and not everywhere else.
users of the low level API may well care about preservation of line endings
What exactly is part of the low level API?
I'm not sure I understand how I preserve line endings in headers in the
current implementation.
"Client code" for LineReaderInputStream should use readLine ONLY when line
recognition is needed (as it already happen).
I have already coded a solution doing this (and using only CRLF and LF as
line delimiters, like the current behaviour).
I'm running a few tests, I'll probably create a JIRA and a proposal
tomorrow.
this proposal seems likely to reduce the correctness and usefulness of
the low level parser in order to address an issue in the high level
API
I'm not sure how can we deal with CR-LF in a consistent way if we don't
do this at a low level, but maybe I'm missing something.
What do you propose should every component in mime4j have the ability to
deal with malformed line endings? Should we make each of them configurable?
ATM I could this places where we deal with newlines:
1) RootInputStream: count lines only when CRLF is found.
2) BufferedLinedReaderInputStream: readLine return byte sequences ending
with LF
3) AbstractEntity: during header parsing ending LF or ending CRLF is
stripped out from lines.
4) QuotedPrintableInputStream deals with isolated \r and \n (not sure
HOW it consider them) and log warning each time they are found isolated.
5) MimeUtil.getHeaderParams consider isolated CR and isolated LF as
newlines (ignoring them) (so that CR is alone is not returned in an
header value)
6) MimeBoundaryInputStream.calculateBoundaryLen seems to strip ending \n
always and \r only if it is before a \n or it is the boundary length is
one more char (BUG? it doesn't recalculate let after the first if)
I don't know if in the javacc/jjtree code we have more CR/LF logic.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
