Rainer: I think...
1. Fragmented messages should be supported over any transport including legacy UDP syslog. If you lose some fragments over unreliable transport, it is expected. A way to determine that this happened would nice to have. I think even your original proposal addresses this with SEQNUM. 2. In the new proposal, why do we need to send message length? The "." indicates the last message, right? Looks like redundant framing methods. 3. I think it is better to talk in terms of message parts (fragments count) instead of length. It is easier for implementers and for human readers. So, I like SEQNUM stuff. 4. Minor thing... Can we start any counts at 1 and not 0? It matches English better -- it is the *first* fragment, not fragment zero. Why reinforce that barrier between CS and normal English? If there is no any IETF conventions that dictate otherwise, I would prefer counts to start 1. (No, I am not a VisualBasic programmer. :)) 5. I am concerned about the scope of reboot id and message sequence numbers? Are they per process? I think we certainly have to support multiple processes firing syslog messages remotely directly without a central daemon. In that case, each has to have its own reboot (restart?) id and message numbers? Then the process MUST be identified in the message, right? There were some discussions on this which I did not quite follow. 6. How do we identify the message to which a given fragment belongs? I can have two processes on the same host originate fragmented messages concurrently. Do all fragments have the MSGNUM of the first message? There has to be some reference in the fragments to the original message. Providing MSGNUM of the first fragment in all subsequent fragments could solve this. 7. I would propose that all fragments of the message should have the same timestamp. Logically, the timestamp should be close to when the event happened, not when it was sent, right? 8. The message length limit is a tough one. I am leaning towards not limiting the length of messages unless we absolutely have to. The receiver should always make sure that it does not attempt to aggregate fragments of messages over what its memory limit (or hard limit) allows it. It is general out of memory error handling which can happen for many reasons, right? Even if we say that message should not exceed X MB, it is likely we would not recommend that implementations just discard messages over this size, but rather default to a different behavior, right? Just like UDP syslog receivers probably won't discard messages over 1024 bytes. 9. Does it make sense to put message fragmentation into a separate RFC from international? The process-unique sequence numbers, for example, can be used in many contexts. 10. Do I think all this is an overkill? Well, what's the alternative? In the interim, at Cisco we had to come up with a pretty similar proprietary scheme. Thanks, Anton. > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Behalf Of Rainer Gerhards > Sent: Tuesday, September 09, 2003 12:02 PM > To: [EMAIL PROTECTED] > Subject: -international - support for oversize messages > > > Hi WG, > > I am preparing a large edit on -international which will include all > those good comments into the draft. I will probably raise a > number of > questions just so that I can verify my understanding of the > discussion. > This is the first of those messages. > > I think we have consensus that there are some (rare) occasions where > oversized messages need to be transmitted via a > standard-compliant way, > thus we need to define something how this can be done. On the other > hand, the vast majority of messages will NOT be oversized, > so we should > add as little overhead as possible to the ID to support this. As a > general guideline, the burden should be placed on those applications > that need to deal with oversized messages while all others > should not > need to pay attention to it. > > syslog-international-00 introduced a very rough way to > implement this. > Here is a quote: > > #### > MORE and SEQNO provide support for syslog messages > larger than the > allowed syslog packet size. This is introduced to allow > transmittal > of "oversized" message, which may be the result of some character > sets and encodings. These messages will be fragmented by > the syslog > client and reconstructed by the collector. Relays will pass them > through unmodifed. > > Message fragmentation MAY be used if the underlying transport > provides reliable and in-order delivery (for example RFC > 3195 [8]). > It the underlying transport is unreliable or its > reliability is not > known, fragmentation MUST NOT be used. > > More specifies wether this is the final fragment of the > message or > not. An asterisk ("*") means that at least one more > fragement will > follow. A period (".") means that this is the final (or only) > fragment. > > SEQNO specifies the sequence number of fragments. It > MUST start by 0 > for the first fragment and MUST be incremented by 1 for each > following fragment. SEQNO MUST restart at 0 for each new full > message. A new full message begins after the last > message that had > "." in MORE. > > If fragementation is not used, all messages contain ". > 0" as the MORE > SEQNO sequence. > #### > > There are a number of issues, especially when it comes to > out of order > delivery. I would like to revise this and would appreciate > feedback on > the following suggestion: > > The modified ABNF would look like this (just the > fragementation-related > parts): > > HDR-i18n = COOKIE SP ENCODING SP CHARSET SP LANGUAGE SP > FRAG-INFO SP > FRAG-INFO = "." / > ("*" SP REBOOTID SP MSGNBR SP TOTALLEN SP INDEX SP > FRAGLEN) > REBOOTID = 1*10DIGIT ; never repeat or decrease > TOTALLEN = 1*7DIGIT > INDEX = 1*7DIGIT > FRAGLEN = 1*3DIGIT > > FRAG-INFO is the important part I am talking about in this > mail. It will > describe if a message is fragmented. > > In the typical case (normal size message), it would just be > ".". So for > most applications, nothing would need to be changed. IF we > have a syslog > implementation that intends to support fragemented oversize package, > THEN (and only then) it needs to provide the following > fields. As you > will probably see, I have borrowed from -sign, which > hopefully will make > it easier to implement: > > REBOOTID is the exact same reboot session ID as in -sign. > An aplication > implementing both -sign and -international SHOULD use the > same value as > in the -sign messages. > > TOTALLEN is the total size of the message in bytes. It probably is > allowed to grow a bit large, but I think it doesn't hurt to support > idiot-size messages. If there is violent opposition, I would like to > limit this to 1 MB, but not less. I have seen *really* > large events > > 100KB once in a year and would like to see these covered > (comparable to > -sign 4.3.5). BUT... I am aware of the fact that a > malicious user could > use this to create a DoS on collectors / log reviewers / > log analysers > that re-assemble the full oversize message. > > INDEX is the position, in bytes, into the oversize message > where this > fragment starts (as is e.g. the fragment index in -sign 4.3.6) > > FRAGLEN is the length, in bytes, of this message. Byte > counting starts > with the byte after the HDR-i18n (that is the byte AFTER > the SP after > FRAGLEN). It can be a maximum of 3 digits because, after > subtracting the > overhead of the headers, no wellformed message (fragement) > can contain > more than 999 bytes. > > If an oversize message is fragmented, the individual > fragments are sent > as syslog messages in their own right, the fragmentation is just > indicated by the proper HDR-i18n fields. All fragments MUST > be send with > the same TAG and HOSTNAME and SHOULD be send over the same > interface. > The TIMESTAMP MUST be set to either the timestamp of the > first fragement > or be increasing (never decreasing). The actual time the message > fragment was scheduled for transmittal SHOULD be placed inside the > TIMESTAMP of each fragment. > > A device supporting oversize messages SHOULD place the most > important > facts early in the message, and SHOULD try to place vital > information > into the first fragment. The reason for this is that a collector or > offline analyzer may decide to discard the message after > some supported > byte count. > > A relay MUST forward the individual (fragmented) messages > unaltered. A > collector SHOULD store them as separate entities. This is > necessary to > verify -sign signatures. Either the collector or an offline > log review > tool can re-assemble the messages. It SHOULD do so before displaying > them to a human or using them for an (automatted) analysis process. > > If fragementation is used, messages SHOULD be sent over a reliable > transport like RFC 3195. No redundancy is provided for > unreliable links, > as this could potentially eat up a lot of bandwidth. > > I think this approach fits the goals. Most importantly, it > allows for > oversize messages without the need to modify any existing > applications, > which also means oversize messages can travel over existing > relays, be > signed with -sign (the individual fragment will be signed) > and also be > used over RFC 3195. > > Is there any opposition against this approach? I am specifically > interested on feedback on: > > - if this is considered overkill compared to the initial approach > - if the approach with using just "." for normal-size messages rings > some bells > - if someone sees some interoperability issues with existing syslog > - if there should be some redundancy be added for > unreliable transport > - as a side note, should the reboot session ID be limited > to the 32 bit unsigned range or left as in -sign? > - if we should include some wording that a collector MUST support > e.g. a message size of 5 KB but MAY discard anything beyond > this. That would resolve the issue with the potential DoS > (at least > most parts of it...). > > I appreciate all feedback! > Rainer > >
